Skip to content

feat: implement filter node executor with support for deduplication, … #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Vamsi-o merged 1 commit into from
Aug 5, 2026
Merged

feat: implement filter node executor with support for deduplication, … #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion apps/web/app/lib/nodeConfigs/filter.action.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ export const filterActionConfig: NodeConfig = {
options: [
{ label: "Remove Duplicates from Provided Data(Single List)", id: "unique_rows" },
{ label: "Find New Data Only (Compare Two Lists)", id: "new_data_only" },
{ label: "Find Existing Data Only (Compare Two Lists)", id: "existing_data_only" }
{ label: "Find Existing Data Only (Compare Two Lists)", id: "existing_data_only" },
{ label: "Group Data By Key", id: "group_by" },
],
Comment on lines 15 to 20
required: true,
defaultValue: "unique_rows",
Expand Down
32 changes: 22 additions & 10 deletions apps/web/app/workflows/[id]/components/ConfigModal.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ export default function ConfigModal({

// Get all tested outputs from Redux (for variable resolution)
const allTestedOutputs = useAppSelector(selectAllOutputs);

const availableNodes = Object.entries(allTestedOutputs).map(([id, output]) => ({
id,
name: output.nodeName || 'Node'
Expand Down Expand Up @@ -364,6 +364,18 @@ export default function ConfigModal({
setConfig(loadedConfig)

const nodeConfig = getNodeConfig(selectedNode.name || selectedNode.actionType);
let finalConfig = { ...loadedConfig };

if (nodeConfig?.fields) {
for (const field of nodeConfig.fields) {
if (field.defaultValue !== undefined && finalConfig[field.name] === undefined) {
finalConfig[field.name] = field.defaultValue;
}
}
}
setConfig(finalConfig);
dispatchConfig(finalConfig);

if (nodeConfig?.fields) {
for (const field of nodeConfig.fields) {
Comment on lines +376 to 380
if (field.fetchOptions && field.dependsOn && loadedConfig[field.dependsOn]) {
Expand All @@ -384,7 +396,7 @@ export default function ConfigModal({
.finally(() => setIsLoadingHeaders(false));
}
}
}, [selectedNode]);
}, [selectedNode?.id]);

if (!isOpen || !selectedNode) return null;

Expand Down Expand Up @@ -436,7 +448,7 @@ export default function ConfigModal({
if (Array.isArray(data[0])) {
return data[0].map((h: any) => String(h));
}

const keys: string[] = [];
const extractKeysRecursive = (obj: any, prefix = "") => {
for (const key in obj) {
Expand All @@ -451,7 +463,7 @@ export default function ConfigModal({
}
}
};

extractKeysRecursive(data[0]);
return Array.from(new Set(keys));
};
Expand All @@ -461,19 +473,19 @@ export default function ConfigModal({

if (field.type === "dynamic_schema_dropdown") {
let options: string[] = ["(No Data Mapped)"];

const dependentFieldName = field.name === "sourceKey" ? "sourceData" : "referenceData";
const dependentFieldValue = config[dependentFieldName];

if (dependentFieldValue && typeof dependentFieldValue === 'string') {
const interpolationContext = buildTestContext();
const resolvedConfig = resolveConfigVariables({ temp: dependentFieldValue }, interpolationContext);
const resolvedData = resolvedConfig.temp;

if (resolvedData !== dependentFieldValue) {
options = extractSchemaKeys(resolvedData);
options = extractSchemaKeys(resolvedData);
} else {
options = ["(Test Node to Load Options)"];
options = ["(Test Node to Load Options)"];
}
}

Expand All @@ -499,7 +511,7 @@ export default function ConfigModal({
let val = opt;
if (opt === "(Value Itself)") val = "__value__";
else if (opt.startsWith("(")) val = "";

return (
<option key={opt} value={val} disabled={opt.startsWith("(") && opt !== "(Value Itself)"}>
{opt}
Expand Down
3 changes: 2 additions & 1 deletion packages/common/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,8 @@ export const FilterNodeInput = z.object({
operation: z.enum([
"unique_rows",
"new_data_only",
"existing_data_only"
"existing_data_only",
"group_by"
]),
sourceData: z.array(z.any()),
referenceData: z.array(z.any()).optional(),
Expand Down
49 changes: 48 additions & 1 deletion packages/nodes/src/filter/filter.executor.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ export class FilterExecutor implements NodeExecutor {
private getValueByPath(obj: any, path: string): any {
if (path === "__value__") return obj;
if (!obj || typeof obj !== 'object') return undefined;

const parts = path.split('.');
let current = obj;
for (const part of parts) {
Expand Down Expand Up @@ -115,6 +115,33 @@ export class FilterExecutor implements NodeExecutor {

return { uniqueData, existingData, discardedData }
}

private handleGroupBy(sourceData: any[], sourceKey: string) {
const groupMap: Record<string, any[]> = {};
let emptyCount = 0;
for (const item of sourceData) {
const key = this.normalizeValue(this.getValueByPath(item, sourceKey))
Comment on lines +119 to +123

if (key === "[EMPTY]") emptyCount++;

if (!groupMap[key]) {
groupMap[key] = []
}
groupMap[key].push(item)
Comment on lines +119 to +130

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Use a prototype-free map for arbitrary group keys.

groupMap is initialized with {}. For a normalized key such as "__proto__", "constructor", or "toString", the lookup returns an inherited object or function. groupMap[key].push(item) then throws, and the executor returns success: false for valid input.

Proposed fix 
-        const groupMap: Record<string, any[]> = {};
+        const groupMap: Record<string, any[]> = Object.create(null);
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
private handleGroupBy(sourceData: any[], sourceKey: string) {
const groupMap: Record<string, any[]> = {};
let emptyCount = 0;
for (const item of sourceData) {
const key = this.normalizeValue(this.getValueByPath(item, sourceKey))
if (key === "[EMPTY]") emptyCount++;
if (!groupMap[key]) {
groupMap[key] = []
}
groupMap[key].push(item)
private handleGroupBy(sourceData: any[], sourceKey: string) {
const groupMap: Record<string, any[]> = Object.create(null);
let emptyCount = 0;
for (const item of sourceData) {
const key = this.normalizeValue(this.getValueByPath(item, sourceKey))
if (key === "[EMPTY]") emptyCount++;
if (!groupMap[key]) {
groupMap[key] = []
}
groupMap[key].push(item)
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nodes/src/filter/filter.executor.ts` around lines 119 - 130, Update
handleGroupBy to initialize groupMap as a prototype-free map so arbitrary
normalized keys such as "__proto__", "constructor", and "toString" are handled
as ordinary groups. Preserve the existing grouping and emptyCount behavior.


}

const groupArray = Object.keys(groupMap).map(key => ({
groupName: key,
rows: groupMap[key]
}))

return {
groupMap, groupArray,
total_processed: sourceData.length,
emptyCount
}
}
async execute(context: ExecutionContext): Promise<ExecutionResult> {
try {
const parsed = FilterNodeInput.safeParse(context.config)
Expand Down Expand Up @@ -171,6 +198,26 @@ export class FilterExecutor implements NodeExecutor {
discardedData = [...existing_data.discardedData, ...existing_data.uniqueData];
break;

case 'group_by':
if (!sourceKey) return {
success: false,
error: "sourceKey is required to group datasets"
}
const groupResult = this.handleGroupBy(normalizedSource, sourceKey);
Comment on lines +201 to +206

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Wrap the group_by switch clause in braces.

Biome reports lint/correctness/noSwitchDeclarations for const groupResult at Line 206. Add a block around this case so the declaration is scoped only to group_by.

Proposed fix 
-                case 'group_by':
+                case 'group_by': {
                     if (!sourceKey) return {
                         success: false,
                         error: "sourceKey is required to group datasets"
@@
                             }
                         }
                     }
+                }
                 default:
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
case 'group_by':
if (!sourceKey) return {
success: false,
error: "sourceKey is required to group datasets"
}
const groupResult = this.handleGroupBy(normalizedSource, sourceKey);
case 'group_by': {
if (!sourceKey) return {
success: false,
error: "sourceKey is required to group datasets"
}
const groupResult = this.handleGroupBy(normalizedSource, sourceKey);
// existing group_by case body
}
default:
🧰 Tools
🪛 Biome (2.5.6)

[error] 206-206: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.

(lint/correctness/noSwitchDeclarations)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nodes/src/filter/filter.executor.ts` around lines 201 - 206, Wrap
the group_by case in the switch statement with braces so the const groupResult
declaration is scoped to that clause, while preserving its existing logic and
return behavior.

Source: Linters/SAST tools


return {
success: true,
output: {
groupsMap: groupResult.groupMap,
groupsArray: groupResult.groupArray,
metadata: {
operation_used: operation,
total_groups: groupResult.groupArray.length,
items_processed: groupResult.total_processed,
items_without_key: groupResult.emptyCount
}
}
}
Comment on lines +208 to +220
Comment on lines +201 to +220

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Align the group_by output contract across both layers.

The executor and node configuration do not publish the same result shape. This can hide grouping results from output discovery or break downstream references.

  • packages/nodes/src/filter/filter.executor.ts#L201-L220: define whether group_by uses an operation-specific shape or the common filter shape, then return the agreed fields.
  • apps/web/app/lib/nodeConfigs/filter.action.ts#L18-L19: add output entries for groupsMap, groupsArray, and the new metadata fields, or make outputSchema operation-aware.
🧰 Tools
🪛 Biome (2.5.6)

[error] 206-206: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.

(lint/correctness/noSwitchDeclarations)

📍 Affects 2 files
  • packages/nodes/src/filter/filter.executor.ts#L201-L220 (this comment)
  • apps/web/app/lib/nodeConfigs/filter.action.ts#L18-L19
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nodes/src/filter/filter.executor.ts` around lines 201 - 220, The
group_by output contract is inconsistent between the executor and node
configuration. In packages/nodes/src/filter/filter.executor.ts lines 201-220,
establish the agreed group_by result shape and return its groupsMap,
groupsArray, and metadata fields; in
apps/web/app/lib/nodeConfigs/filter.action.ts lines 18-19, expose those same
fields by adding output entries or making outputSchema operation-aware. Keep
both layers aligned so grouping results are discoverable and downstream
references resolve correctly.

default:
return { success: false, error: `Unknown operation: ${operation}` };
}
Expand Down