Async loading of fields to select field based on the keyword that typing in

[shijiezhou1]

The react-awesome-query-builder not able to provide the option to async load the selecting field.

if i have more than 5000 fields, the program will stuck on hanging on the screen.

see the code below

const fields = {};
    for (let i = 0; i < 1000; i++) {
        fields[`qty_${i}`] = {
            label: `Qty ${i}`,
            type: 'number',
            tooltip: "Qty",
            fieldSettings: {
                min: 0,
                fetchSelectedValuesOnInit: true
            },
            valueSources: ['value'],
            preferWidgets: ['number'],

        };
        fields[`price_${i}`] = {
            label: `Price ${i}`,
            type: 'number',
            valueSources: ['value'],
            fieldSettings: {
                min: 10,
                max: 100,
            },
            preferWidgets: ['slider', 'rangeslider'],
        };
        fields[`name_${i}`] = {            label: `Name ${i}`,
            type: 'text',
        };
        fields[`color_${i}`] = {
            label: `Color ${i}`,
            type: 'select',
            valueSources: ['value'],
            fieldSettings: {
                listValues: [
                    { value: 'yellow', title: 'Yellow' },
                    { value: 'green', title: 'Green' },
                    { value: 'orange', title: 'Orange' }
                ],
            }
        };
        fields[`is_promotion_${i}`] = {
            label: `Promo? ${i}`,
            type: 'boolean',
            operators: ['equal'],
            valueSources: ['value'],
        };
    }

    const emptyInitValue = { id: Utils.uuid(), type: "group" }
    const [initConfig, setInitConfig] = useState({ ...MuiConfig, fields })

    const [state, setState] = useState({
        tree: Utils.checkTree(Utils.loadTree(emptyInitValue), initConfig),
        config: initConfig
    })


 <Query
                {...state.config}
                value={state.tree}
            />

how can i make it work? so i can load 5000 fields async? or can we use react-window or virtual scrolling package? any solution?

Thank you

[ukrbublik]

Need to consider using react-window or virtual scrolling

[ukrbublik]

Do you use Antdesign?

[ukrbublik]

Possibly related: #1209

[perpetual-cosmos]

Hi! While I haven’t worked directly on async loading or virtualization in react-awesome-query-builder before, I’m eager to research and learn the best solutions. I’m excited to collaborate and contribute to making the component handle thousands of fields smoothly.

Looking forward to working together—thanks for considering me!

[perpetual-cosmos]

Hi @shijiezhou1 @ukrbublik ,

I’ve reviewed the issue and CONTRIBUTING.md, and I’d like to propose a solution to enable async field loading to handle large field sets (e.g., 5000+ fields) efficiently.

Proposed Solution:

1. Add an asyncFetch function to the config’s settings to filter fields dynamically based on user input:

   settings: {
     asyncFetch: async (search) => {
       const fields = await fetchFields(search); 
       return { fields, total: Object.keys(fields).length };
     }
   }

• Is asyncFetch already supported in the library’s config? If not, should I add it to packages/core or packages/ui?
• Which component handles the field selection dropdown? I searched packages/ui/modules/components but couldn’t find the exact file.
• Should I start with the demo app (packages/examples) or modify the library directly?
• Are there specific guidelines for async support (e.g., preferred libraries or patterns)?

I’ve set up the repo locally using Git Bash on Windows (resolved sh error during pnpm i) and can test changes in the demo app. Any pointers on where to implement this or specific files to check would be helpful!

[ukrbublik]

Is asyncFetch already supported in the library’s config?

It has a different purpose - to load values for field dynamically (eg. see field "Autocomplete" in demo app)

react-awesome-query-builder/packages/examples/src/demo/config/index.tsx

Line 793 in 1035fd7

asyncFetch: simulatedAsyncFetch,

If not, should I add it to packages/core or packages/ui?

If you want to use approach with using asyncFetch (or similar function name) for fields it's better to put it in core.
But I think it's complex to load fields dynamically cause initially the lib was designed to use fields directly from config.fields
Did you consider using virtual scrolling instead?

Which component handles the field selection dropdown?

It depends on UI framework.
For basic UI - VanillaFieldSelect
For MUI - MuiFieldAutocomplete or MuiFieldSelect
Controlled by renderField in config

react-awesome-query-builder/packages/mui/modules/config/index.jsx

Line 10 in 1035fd7

renderField: (props, {RCE, W: {MuiFieldAutocomplete, MuiFieldSelect}}) => props?.customProps?.showSearch

Should I start with the demo app (packages/examples) or modify the library directly?

I suggest first adding code to mimic high number of fields (see sample code to generate 1000 fields in first comment by @shijiezhou1) in demo app and see how it handles large number of fields, then try to use virtual scrolling by modifying library, or apply other rendering optimizations.

[perpetual-cosmos]

Hi @ukrbublik,

Thank you for your guidance on issue! I’ve reviewed packages/examples/src/demo/config/index.tsx ), packages/mui/modules/config/index.jsx, and packages/core/modules/config/default.js. I understand that asyncFetch is for field values, and you recommend virtual scrolling to handle large field sets (5000+ fields).

Proposed Solution:

1. Add the 5000+ fields from the issue’s sample code to packages/examples/src/demo/config/index.tsx to replicate the performance issue.
2. Implement virtual scrolling in the field dropdown:
   • For MUI, modify MuiFieldAutocomplete (or MuiFieldSelect) in packages/mui/modules/components/field to use @mui/material/Autocomplete with react-window’s FixedSizeList for virtualization.
   • For Vanilla UI, update VanillaFieldSelect in packages/ui/modules/components/field to use react-window.
3. Ensure renderField in packages/mui/modules/config/index.jsx uses the virtualized component when customProps.showSearch is true.
4. Test performance in the demo app (pnpm start) with the large field set.
5. If virtual scrolling isn’t sufficient, explore it further.

Questions:

• Should I prioritize virtual scrolling for MUI (MuiFieldAutocomplete/MuiFieldSelect) or Vanilla (VanillaFieldSelect)?
• Is react-window a good choice for virtual scrolling, or do you prefer another library (e.g., react-virtualized or MUI’s built-in virtualization)?
• I couldn’t find MuiFieldAutocomplete.jsx, MuiFieldSelect.jsx in packages/mui/modules/components/field, or VanillaFieldSelect.js in packages/ui/modules/components/field. Could you confirm their exact paths or names?
• For testing, I’ll add the 5000+ fields to config/index.tsx. Is this the best place, or should I use another file in packages/examples?

I’m running the repo locally with Git Bash on Windows (pnpm start works after resolving the sh error). I’ll start by adding the large field set to the demo app and testing the dropdown performance. Any clarification on the component file paths or virtual scrolling preferences would be very helpful!

[ukrbublik]

Should I prioritize virtual scrolling for MUI or Vanilla ?

I don't think vanilla should be changed at all. Priorities are MUI and AntDesign (packages/antd/modules/widgets/core/FieldSelect.jsx, packages/antd/modules/widgets/core/FieldTreeSelect.jsx, packages/antd/modules/widgets/core/FieldCascader.jsx)

Is react-window a good choice for virtual scrolling?

I didn't work with virtual scrolling libraries, but I think that built-in virtualization is preferred - MUI uses react-window so we should follow the example. Please investigate what technique is used in AntDesign but I think virtualization is enabled by default (see virtual prop)

Could you confirm their exact paths or names?

https://github.com/ukrbublik/react-awesome-query-builder/blob/master/packages/mui/modules/widgets/core/MuiFieldSelect.jsx
https://github.com/ukrbublik/react-awesome-query-builder/blob/master/packages/mui/modules/widgets/core/MuiFieldAutocomplete.jsx
https://github.com/ukrbublik/react-awesome-query-builder/blob/master/packages/antd/modules/widgets/core/FieldSelect.jsx
https://github.com/ukrbublik/react-awesome-query-builder/blob/master/packages/ui/modules/components/widgets/vanilla/core/VanillaFieldSelect.jsx

Is this the best place, or should I use another file in packages/examples?

I would suggest adding new file (named smth like generatedBig.ts ) in https://github.com/ukrbublik/react-awesome-query-builder/tree/master/packages/examples/src/demo/config that exports a function to generate big config.
Next add a button like "Use big config" next to "Benchmark: " at top.
(see https://github.com/ukrbublik/react-awesome-query-builder/blob/master/packages/examples/src/demo/blocks/benchmark.tsx)
Action on this button should update demo app state with big config.
Also please add a button to restore original config as well.

pnpm start works after resolving the sh error

What error did you have? Sorry, I'm not a Win user.