Skip to main content

Developer Guide

Whether you're building custom extensions, contributing to the project, or integrating Viglet Dumont into your CI/CD pipeline, this guide has everything you need.

Dumont is fully open-source at github.com/openviglet/dumont. All contributions are welcome.

Tech Stack​

LayerTechnology
BackendJava 21 · Spring Boot 4.0.3
Message QueueApache Artemis (embedded)
DatabaseH2 (dev) · PostgreSQL (prod)
HTML ParsingJSoup 1.22.1
Text ExtractionApache Tika 3.2.3
Search ClientsTuring Java SDK · SolrJ · ES Client
ConsoleReact 19 · TypeScript · Vite · Tailwind CSS v4
Console components@viglet/viglet-design-system
BuildApache Maven (backend) · pnpm (console)
CI/CDGitHub Actions

Setting Up Your Dev Environment​

Prerequisites​

  • Java 21 (Temurin recommended)
  • Maven 3.9+
  • Git
  • Viglet Turing running at http://localhost:2700 (for end-to-end testing)
  • Node.js 26 and pnpm 11, to work on the console (the Maven build invokes them for you)

Clone, Build, and Run​

git clone https://github.com/openviglet/dumont.git
cd dumont
mvn clean install

cd connector/connector-app
mvn spring-boot:run

The application starts at http://localhost:30130.

Project Structure​

dumont/
├── commons/                    # Shared interfaces, models, utilities
├── aem-commons/                # AEM extension interfaces (published to Maven Central)
├── spring/                     # Spring Boot configuration and JPA persistence
├── connector/
│   ├── connector-commons/      # Connector SPI: structured sources, schema, grounding, enrichment
│   └── connector-app/          # Main pipeline: strategies, batch, queue, indexing plugins, API
├── web-crawler/
│   └── wc-plugin/              # Web Crawler connector plugin
├── db/
│   ├── db-commons/             # DB extension interface (published to Maven Central)
│   ├── db-app/                 # Database connector (standalone CLI)
│   └── db-sample/              # Example custom DB extension
├── filesystem/
│   └── fs-connector/           # FileSystem connector (standalone CLI)
├── aem/
│   ├── aem-plugin/             # AEM connector plugin (loaded via -Dloader.path)
│   ├── aem-server/             # AEM server-side integration
│   └── aem-plugin-sample/      # Example custom AEM extensions (WKND site)
├── dumont-react/               # The console: React app, built into connector-app's static resources
└── wordpress/                  # WordPress PHP plugin

Extension Points​

Dumont provides extension points at multiple levels:

Connector-Level Extensions​

ExtensionGuideMaven Artifact
AEM extensions: attribute extractors, model.json processors (with fluent API), delta date logicExtending the AEM Connectorcom.viglet.dumont:aem-commons:2026.2.3
Database extensions: custom row transformations during SQL importExtending the Database Connectorcom.viglet.dumont:db-commons:2026.2.3

Platform-Level Extensions​

Creating a Custom Connector​

Most sources are worth modelling for their structure (a listing plus per-item detail pages with a typed schema) rather than as a link spider. For those, implement DumStructuredSource: its identity, its schema, and the two methods that read the source.

public class CourseCatalog implements DumStructuredSource {

    private final DumStructuredSchema schema = DumStructuredSchema.builder()
            .field(DumStructuredField.builder("title").type(TurSEFieldType.TEXT)
                    .mandatory(true).build())
            .field(DumStructuredField.builder("campus").type(TurSEFieldType.STRING)
                    .facet(true).build())
            .build();

    @Override public String getSourceName()            { return "courses"; }
    @Override public String getProviderName()          { return "CATALOG"; }
    @Override public Collection<String> getSiteNames() { return List.of("Sample"); }
    @Override public DumStructuredSchema schema()      { return schema; }

    @Override public List<String> discover()           { /* every content id */ }
    @Override public Optional<DumStructuredRecord> extract(String id) { /* one record */ }
    // getLocale() defaults; override it when the source is not en-US.
}

Then extend DumStructuredSourcePlugin, which supplies the whole discover → extract → enrich → publish loop, and with it everything the framework already does for every structured source:

You getWhat it does
Schema as codeThe schema is the single declaration of every field. It becomes the attribute specs published with each document, and it is sent to the backend before the first document so fields exist as declared instead of being inferred from the first value seen
Conservative groundingA null, blank or empty value is dropped at the record boundary rather than indexed. A field is absent or it has a value, never "" in a facet
EnrichmentA chain of DumEnrichers runs between extract and publish, merging through the schema-bound builder so enrichment cannot break grounding, and a failing enricher is isolated
Safe partial runsAn interrupted or failing run finishes standalone, skipping the deindex sweep, so the previous index survives instead of the un-visited ids being treated as removed
Coverage and driftPer-field coverage is recorded per run, so a layout change that silently stops extracting a field is reported instead of discovered months later
Dry-run previewA run can be previewed against the last full index's fingerprint before anything is published

If your source genuinely is not structured (an event-driven push, a protocol of its own), the low-level interface is still there:

public interface DumConnectorPlugin {
    void crawl();
    String getProviderName();
    void indexAll(String source);
    void indexById(String source, List<String> contentId);
}

Creating a Custom Indexing Plugin​

Implement the DumIndexingPlugin interface:

public interface DumIndexingPlugin {
    void index(TurSNJobItems turSNJobItems);
    String getProviderName();

    // Both default to doing nothing, so a backend that cannot do them is unchanged.

    /** Declare a source's field schema before its documents. */
    default void provision(DumIndexingSchema schema) { }

    /** Report what the index holds for a source, for the content audit's comparison.
     *  Empty means "cannot answer", distinct from a present, empty list. */
    default Optional<List<DumIndexInventoryEntry>> inventory(String siteName, String source) {
        return Optional.empty();
    }
}

Register your plugin as a Spring @Component with @ConditionalOnProperty(name = "dumont.indexing.provider", havingValue = "your-provider").

Processing Strategy Architecture​

Strategies are evaluated in priority order for each Job Item:

Dumont, Processing Strategy Flow

To add a custom strategy, implement the strategy interface and assign a priority between the existing ones.

Working on the Console​

The console lives in dumont-react/. Maven builds it as part of mvn clean install and copies the output into connector-app's static resources, so a normal backend build already produces a working UI. Work on it directly when you want hot reload:

cd dumont-react
pnpm install
pnpm run dev          # Vite dev server
pnpm test             # Vitest
pnpm run lint         # ESLint
pnpm run lint:ds      # see "Shared components" below
pnpm run census       # which surfaces are on which chrome

What CI checks​

Two workflows gate every push and pull request:

WorkflowChecks
validateThe Maven build, which also builds the console, and a SonarCloud analysis on pushes
design-systemThat no component re-declares a design system export, and ESLint

Two details worth knowing before a build surprises you:

  • ESLint has no exemptions. It arrived on a console that had never been linted, and three rules carried the debt: @typescript-eslint/no-explicit-any, react-hooks/set-state-in-effect and react-hooks/exhaustive-deps. Each was held for a while as a list of named files rather than a tolerated count, and each list was emptied and then deleted. All three are errors everywhere now. If a rule needs that treatment again, name the files in eslint.config.js and say why at each one; never raise a ceiling, because a ceiling lets one change pay for another. A genuine one-off exception is an eslint-disable-next-line with its reason on the line: there is exactly one, in use-grid-adapter.ts.

  • set-state-in-effect almost always means the value is not state. Clearing that rule across the console took twenty-five fixes and produced two shapes, both of which delete the write rather than move it:

    • Derive it. Most of them were facts, not state: whether the route is a new record, a blank entity the render can compute, which tab the URL names.
    • Store the answer against the question. A result kept together with the query that produced it makes "loading" derivable (you are waiting while the answer in hand is not the answer to what is being asked now) and the resets unnecessary. It also removes stale renders, which is how it found two real bugs: an answer to a previous query can no longer be read as this one's.

    Do not wrap a synchronous write in void (async () => { … })(). The rule cannot see through it, the write is still synchronous, and that is a loophole rather than a fix. Calling an async loader that way is accepted: void (async () => { await load() })().

  • The console build ends by checking that the federated remotes declared their types. The generator reports failure badly (it has printed an error and exited 0, and it has silently written the wrong layout and exited 0), so scripts/check-mf-types.mjs inspects the artefact instead of trusting the exit code.

The console runs in two places. Standalone it is served by the connector itself; inside Viglet Turing it is mounted as a Module Federation remote, and Turing supplies the surrounding chrome. Pages are written once and work in both: a fixed local integration id stands in for the one Turing would pass, and an axios interceptor strips it from API calls.

Shared components​

Buttons, dialogs, tables, form controls and page chrome come from @viglet/viglet-design-system, the component library Dumont shares with Viglet Turing and Viglet Shio. Import from it rather than writing a local copy: a copy drifts, and the three products stop looking like one.

CI enforces this. pnpm run lint:ds fails the build when a component is declared here under a name the design system already exports, and each finding names the import that replaces it. Where a collision is deliberate (Dumont's AppFooter renders Dumont's own version and links, and merely shares a name), annotate the declaration with the reason:

// viglet-ds-allow-duplicate AppFooter -- renders Dumont's own logo, connector version and links

Two generations of chrome​

Console surfaces are being moved to bento (the design system's newer visual language, the same one Turing uses), one page at a time. Both generations serve at once:

TreePathChrome
Console/admin/integration/instance/:id/…Collapsible sidebar
Bento/bento/integration/instance/:id/…Fixed nav rail, ⌘K command palette

A page appears under /bento only once it has actually moved; its /admin route keeps working either way, so nothing breaks while the migration is in progress and a page can move back. Both trees read one navigation declaration (src/app/nav.const.ts), which is also what the command palette searches, so the sidebar, the rail and the palette can never disagree about which surfaces exist.

Every source surface has moved. All six connector types (AEM, database, web crawler, filesystem assets, catalog bridge and Edge Delivery Services) list and edit under /bento, so configuring a source end to end never leaves the new shell. The remaining console surfaces (indexing rules, monitoring, statistics, coverage, plugins, system information) are reached from the rail's section hubs or ⌘K, which link out to /admin until they move too.

Migrating a page​

Run pnpm run census to see which surfaces are on which era, then:

A list page is one BentoSourceList call: the shape is shared, so a new one supplies its service, two field accessors and a translation prefix. Its path, icon and tone come from the navigation declaration by surface id, so a page cannot disagree with the rail about what it is called.

A detail page does not get rewritten. Its form is hundreds of lines of field logic whose only era-specific parts are the header and the section cards, so those read the chrome instead:

// The page: the console page under a chrome declaration.
<SectionCardChromeProvider chrome="bento">
  <IntegrationInstanceDbSourcePage />
</SectionCardChromeProvider>

Inside the form, AdaptiveFormHeader renders the console's sticky header or the bento hero and its save-bar morph, and AdaptiveSectionCard renders a collapsible card or a frosted section. Both default to console chrome, so a form that knows nothing about this behaves exactly as it always did, and one form serves both eras rather than being forked into two that drift.

Finally set migrated: true on the surface in nav.const.ts. That is what gives it a bentoRoute, which is how the palette and the hubs know to stay inside the shell instead of linking out.

REST API​

EndpointDescription
GET /api/v2/connector/statusHealth check
POST /api/v2/connector/indexing/Submit indexing jobs
GET /api/v2/connector/monitoring/index/{source}Monitor indexing progress
GET /api/v2/connector/validate/{source}Validate a content source

For the full API surface, start the application and visit the Swagger UI.

Contributing​

  1. Fork the openviglet/dumont repository
  2. Create a branch for your feature or fix: git checkout -b feature/my-improvement
  3. Commit with clear, descriptive messages
  4. Open a Pull Request: describe what you changed and why
ON THIS PAGE