Without this fix, a DbArray of List<Integer> on the plaforms that do NOT natively
support it like MySQL, Oracle, SQL Server fallback to a JSON array and this didn't
support all the specific types like Integer.
This adds the appropriate element converters when reading values, including Integer, Long, floating-point, BigDecimal, date/time, and UUID types. This prevents JSON numeric values from being returned as Long for List<Integer> properties.
* Publish timed metric max values over rolling 59-second windows
* Publish timed metric max values over rolling 59-second windows
Use "now - 2 * WINDOW_NANOS" on initialise and reset to ensure
the next collection publishes.
---------
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
* Add test for #1852 : cascaded delete order broken by a write from a persist callback
A join entity owns the foreign key to the bean its delete cascades to, so the join row
has to be deleted first. When a BeanPersistController writes to the database from
preDelete, that write flushes the batch from inside the flush that is already running :
the outer flush has taken the join rows out of their bean holder, so the inner flush
finds only the assets and executes them first, which fails on the foreign key.
BatchControl flush [DcoLink:0 d:2, DcoAsset:1 d:2] <- outer flush
BatchControl flush [DcoLink:0 d:0, DcoAsset:1 d:2, DcoAudit:2 i:1] <- from preDelete
The scenario is fixed as a side effect of a2f954a60 (#3830, released in 18.3.0) which
moved controllerPreDelete() ahead of the cascade. The test pins that down : it fails
with a DataIntegrityException on a2f954a60~1 and passes on master. The graph is fetched
up front on purpose, a lazy load would flush the batch on its own and hide the ordering.
* Add failing test reproducing #1852 : out of order cascaded delete on a self referencing tree
The shape reported on the issue in 2019 : a container cascades the delete down a tree of
TreeBean, and the deletes are not issued deepest first. On 18.4.0 :
delete from dco_tree where id in (?) -- the root, whose children are still there
delete from dco_tree where id in (?,?,?)
delete from dco_tree where id in (?,?)
Referential integrity constraint violation: FK_DCO_TREE_PARENT_ID.
This is a different defect from the batch reordering fixed by a2f954a60 : nothing is batched
here, the recursion itself walks the tree in the wrong order. Disabled so it does not break
the build, remove the annotation to see the failure.
* FIX: a persist done from a BeanPersistController callback flushes the batch mid-execution
#3148 stopped a query performed from a callback from flushing the batch that is already
executing : BatchControl.executeNow disables flushOnQuery for the duration. A persist done
from the same callback is not covered. It reaches BatchControl.executeOrQueue, which flushes,
and the statements queued behind the one currently executing are issued early.
Saving a parent/child graph in batch while an audit row is written from preInsert issues the
children before their parent has an id :
insert into dco_link (parent_id, asset_id) values (?,?)
NULL not allowed for column "PARENT_ID"
Same defect on the delete side, where the join rows are issued after the beans they reference
(#1852, #3185) — that path no longer reproduces since a2f954a60 moved controllerPreDelete()
ahead of the cascade, but only preDelete was moved, so preInsert and preUpdate still run
inside the flush.
Guard executeOrQueue with the same reasoning as the existing flushOnQuery guard : while the
batch is executing, queue rather than flush. The statements added meanwhile are picked up by
the do/while loop in executeAll().
* FIX: same guard on executeStatementOrBatch, a SqlUpdate from a callback also flushes mid-execution
executeStatementOrBatch() flushes on (batchFlushOnMixed && !isBeansEmpty()), and persistedBeans
is only cleared once executeAll() returns, so during the batch execution that condition holds and
a SqlUpdate run from a BeanPersistController callback re-enters the flush the same way a save does.
Reproduced with the same test, the callback running a SqlUpdate instead of a save :
insert into dco_link (parent_id, asset_id) values (?,?)
NULL not allowed for column "PARENT_ID"
The second flush of that method, on pstmtHolder.maxSize() >= batchSize, is left untouched : it can
only trigger when the pstmt holder fills up mid-execution and there is no test covering it.
Read-only TxScope previously ignored isolation when creating
ImplicitReadOnlyTransaction. Apply setIsolationLevel after
createReadOnlyTransaction so @Transactional(readOnly=true, isolation=...)
and TxScope setReadOnly+setIsolation take effect.
Co-authored-by: arimu1 <19286898+arimu1@users.noreply.github.com>
* Metrics - support both CUMULATIVE and DELTA metrics collection concurrently
ebean insight [and StatsD] work off DELTA metrics, where as OTEL
and Prometheus want CUMULATIVE. With this change we can have a metrics
collection for ebean insight using DELTA mode and have a second collection
use CUMULATIVE for reporting to OTEL - for the case of fan-out metrics
going to 2 places.
This isn't strictly needed when only one collection mode is used.
* Metrics - Add Mode with RESET, DELTA and CUMULATIVE
Previously we were overloading RESET and DELTA but we really need
these to be 2 separate modes for the existing tests and the
get(reset) api
* Fix test TestNatKeyCacheWithForeignKey with cache stats reset
---------
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
Adding this as I am seeing a LOT of cases where ebean can derive a good enough
error message (id case, plus single unique key case) that it is worth adding
this syntax sugar.
Yes we have discussed this a well it's not so useful for the multi-lingual /
non-english case and also marginal when the error message needs to be built
case.
When a *ToOne association is fetched with only its id property, the
foreign key column already exists on the owning table so no join is
required. Previously this always generated an unnecessary join.
- OrmQueryProperties: add includesExactly() and withAddedInclude() to
support adding an include without mutating a shared/cached instance
(the included set and its precomputed hash prefix are immutable
since #3761/#3763, so an "add" must produce a new instance).
- OrmQueryDetail: add convertIdFetches(), a bottom-up pass over
fetchPaths that replaces an id-only *ToOne fetch with its FK
property folded into the parent's select, removing the now-unneeded
fetch path. Skips exported (mappedBy) one-to-one associations, which
have no local FK column and always require a join. Tracks paths
still depended on by a surviving child fetch so their join isn't
incorrectly folded away.
- Add TestFetchIdOnly covering PathProperties, select(), and fetch()
usages, plus a control case confirming the join is retained when
more than the id is fetched.
- Update EqlParserTest, TestSubQuery and TestMergeCustomer assertions
to reflect the new join-free SQL.
* Version 18.3.0
* InsertOnConflict - change to exclude non-updatable and generated on insert only (e.g. @WhenCreated)
---------
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
* #2540 Add DTO mapping from Entity beans via generated mapper code
A large feature, refer to the docs / guides etc. This is a bit like MapStruct in that it generates mapping code via annotation processing but it ALSO creates an optimised FetchGroup for the destination DTO graph and it also implements a generic interface such that it can be fluently used via query.mapTo().
The value is that it automates the mapping code from entity -> dto plus optimises the query used to support building the DTO graph.
Expected to be better to use than MapStruct for ebean orm users.
* Bump ebean-annotation to 8.6 with the dto annotations
* Adjust the build order to via test dependency on ebean-api
---------
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
* Bug: filterMany() expressions when additional nested fetch has predicates in wrong place
When a filterMany() expression only references the many-root's own properties (e.g. .eq("status", ...)),
but the query also has an additional nested fetch beneath that many-path (e.g. .fetch("orders.customer")
or .fetch("contacts.group")), the filter predicate was incorrectly attached to the last/deepest join's
ON clause instead of the many-root's own join — silently misapplying the filter.
Fix: A hybrid approach:
- If the filterMany expression references only root-level properties → attach the predicate directly on the many-root's own join (correct).
- If it references deeper/nested properties requiring "extra joins" (a separate existing mechanism) → fall back to the original end-of-subtree behavior, since no SqlTreeNode exists to attach to.
- includeFilterMany() made idempotent so both mechanisms coexist safely.
* Bug: filterMany() with nested-property predicates silently ineffective
filterMany() expressions referencing a nested/dotted property (e.g.
.eq("group.name", "x"), alone or mixed with root properties) were
attached to a LEFT JOIN's ON clause via the "extra join" mechanism.
That only nulls the extra join's own columns on non-match - it does
not exclude the parent/many-side row, so the predicate had no actual
filtering effect (confirmed by data, not just SQL shape).
## Fix:
force any many-property fetch whose filterMany expression
references a nested property onto a query-join (secondary select
restoring filterMany's original always-fetchQuery behaviour for this
with a genuine WHERE clause) instead of leaving it as an inline JOIN, case.
- SpiExpressionValidation: track all visited property names (allProperties()), not just unknown ones.
- OrmQueryProperties: filterManyHasNestedProperty() walks the filterMany expression for any dotted property reference.
- OrmQueryDetail.markQueryJoins(): route nested-property filterMany chunks to markForQueryJoin() instead of the inline fetch-join slot, without consuming that slot for other many-paths.
Also simplifies the earlier root-only-predicate join-placement fix
(CQueryPredicates/DefaultDbSqlContext/SqlTreeNodeManyRoot) now that
a nested-property filterMany can never reach the JOIN-based code
path: removed the now-unreachable "deepest path" fallback branch,
the includeFilterMany() idempotency guard, and the redundant fallback
call in SqlTreeNodeManyRoot - isFilterManyAttachPoint() remains as
the single, always-used attach mechanism.
Updated TestQueryFilterMany/TestQueryFilterManySimple assertions to
expect the corrected query-join (2 statement) behaviour.
* Simplify SpiExpressionValidation with boolean nestedProperty flag
* Tidy up comments
* Fix OrmQueryProperties.filterManyHasNestedProperty() with real nested property check
---------
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
SQL Server does not support a WITH clause (CTE) nested inside a subquery
or derived table - only at the start of a statement. buildRowCountQuery()
and buildExistsQuery() wrap raw sql as `select count(*) from (<sql>)` /
`select case when exists(<sql>) then ...`, which breaks when <sql> starts
with a CTE header (e.g. RawSqlBuilder.withPlaceholders() queries), causing
"Incorrect syntax near the keyword 'with'" on SQL Server.
Add CQueryBuilder.topLevelSelectStart()/splitCteHeader() to detect and
hoist a leading WITH clause in front of the wrapping SELECT rather than
wrapping it along with the rest of the query. This is a no-op for the
common case (no leading CTE) and is portable across platforms.
- ebean-core: CQueryBuilder - hoist leading CTE header in
wrapSelectCount()/wrapSelectExists()
- Add unit tests in CQueryBuilderTest reproducing the exact failing SQL
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
* #3848 Fix regression for Sql Server and Oracle with exists() introduced in 18.2.0
Fix#3848: exists() generates invalid scalar SQL on SQL Server and Oracle
Query.exists() generated `select exists(<subquery>)`, which is valid on
Postgres/H2/MySQL but rejected by SQL Server and Oracle since both only
support EXISTS as a predicate, not as a directly selectable scalar boolean
expression. ┃
┃
Add DatabasePlatform.existsWithCaseWhen/existsFromClause capability flags
and use them in CQueryBuilder.wrapSelectExists() to generate
`select case when exists(<subquery>) then 1 else 0 end` on platforms that ┃
need it, with an additional ` from dual` suffix for Oracle (which requires ┃
a FROM clause on every select). CQueryExists reads the boolean result via ┃
ResultSet.getBoolean(1), which correctly interprets the resulting 0/1 int.
* Fix tests TestInsertCheckUnique for Oracle and SQL Server
---------
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
## Root cause:
For @IdClass composite keys, the "id" is cached only in EntityBeanIntercept.ownerId(), populated only when reading from DB or during insert derivation. A fresh bean instance passed straight to DB.update()/DB.delete() had this null, causing an NPE in BindableIdEmbedded.dmlBind().
## Fix:
dmlBind() now defensively derives the id (reusing the existing insert-time derivation logic, extracted into a shared deriveId() helper) when it's null but derivable — covering both update and delete, since they share the same DML bind path.
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
## Root cause:
Finder.byId() on a @Cache entity converts the id to a String cache key via IdBinder.cacheKey() = scalarType.format(value). For enums, format() returns .name() (e.g. "APPROVED"), not the db-mapped value. On a cache hit, BeanDescriptorCacheHelp.loadBeanDirect() converts that cache-key string back via desc.convertId() → IdBinderSimple.convertId() → scalarType.toBeanType(), which expects a db value (e.g. "2" for @EnumValue("2")), not the enum's Java name — causing Integer.parseInt("APPROVED") to crash in EnumToDbIntegerMap.getBeanValue().
## Fix:
IdBinderSimple.convertId() now special-cases String input to use scalarType.parse() (the true inverse of format()) instead of toBeanType() (the inverse of toJdbcType()/db-value). This mirrors an existing, already-correct pattern in IdBinderEmbedded.convertId(), which already handles String cache-key round-tripping via parse(). Verified this doesn't affect other id types (Long/Integer/etc.) since parse() and toBeanType() are equivalent for them — the mismatch only exists for Enum-mapped types.
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
## Root cause:
AnnotationAssocManys only parsed @OrderColumn inside the @OneToMany branch, and even if parsed, BeanDescriptorManager.checkMappedByOneToMany() returned early for element collections before reaching the order-column setup. The element collection's synthetic descriptor also isn't registered in deployInfoMap, so it couldn't reuse the existing @OneToMany lookup path.
## Fix (in ebean-core):
- BeanDescriptorManager: extracted makeOrderColumn(DeployBeanPropertyAssocMany, DeployBeanDescriptor) so the synthetic order property can be built against an explicit target descriptor, not just one looked up via deployInfoMap.
- AnnotationAssocManys.readElementCollection(): now reads @OrderColumn, sets fetchOrderBy, and builds the order property directly on the element descriptor before it's converted to a runtime BeanDescriptor.
- SaveManyElementCollection: binds a sequential 0-based index as the order column value on insert (element collections delete-then-reinsert the whole collection in list order, so this is a natural fit).
DDL generation and the fetch order by needed no changes - both already generalize over any BeanDescriptor's order column property once it exists.
Added EcolPerson / TestElementCollectionOrderColumn covering insert population, order preserved on reload, and order by present on fetch-join SQL.
Known limitation: @OrderColumn on a Map-based element collection (SaveManyElementCollectionMap) is not wired up - not part of this issue and not a standard JPA combination (Maps use @MapKeyColumn).
Root cause: The built-in ScalarTypeJsonMap/List/Set types (used for Map<String,Object>, simple List/Set) hard-coded mutable()=true and isDirty() to always do a ModifyAware check, regardless of the configured MutationDetection. Only SOURCE accidentally worked because it happened to route through BeanPropertyJsonMapper.
Fix (in ebean-core, io.ebeaninternal.server.type):
- ScalarTypeJsonValue now derives mutable (false only for NONE) and keepSource/jsonMapper() (true for HASH/SOURCE) from the property's MutationDetection, instead of a single hard-coded keepSource boolean.
- ScalarTypeJsonMap, ScalarTypeJsonMapEnum, ScalarTypeJsonList, ScalarTypeJsonSet, and ScalarTypeJsonCollectionValue now accept/pass through MutationDetection instead of a raw boolean.
- DefaultTypeManager.dbJsonType() passes the property's own mutationDetection() straight through for these collection types (per your choice: DEFAULT still always means ModifyAware for collections — only an explicit NONE/HASH/SOURCE annotation changes behavior; the DatabaseConfig-wide default is not applied to these types).
- @DbArray fallback call sites (PlatformArrayTypeJsonList/Set) updated to pass MutationDetection.DEFAULT, preserving unchanged behavior.
On standards-based platforms (MariaDB, SQL Server, Oracle, DB2, HANA) the root table always generates a 'for system_time between ? and ?' clause for TemporalMode.VERSIONS, but bind values were only supplied for findVersionsBetween(), leaving findVersions() with 2 unbound placeholders and shifted bind positions ("Parameter at position N is not set").
Match the bind guard to the SQL generation condition and default start/end to epoch/now when not explicit.
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
* checkUniqueness supports queryCache and skipClean
* Keep the DB.checkUniqueness() as per original (not overload there)
---------
Co-authored-by: Roland Praml <roland.praml@foconis.de>
* Properties of inherited models can be used on different child models
* Adjust typos only
---------
Co-authored-by: Roland Praml <roland.praml@foconis.de>
* Add RawSqlBuilder.withPlaceholders() for CTEs and other complex SQL (#3649)
RawSqlBuilder.parse() uses keyword-based scanning to locate SELECT
columns and the WHERE/HAVING injection points. This fails for CTEs,
window functions and other complex SQL where "select"/"from" keywords
appear in places the scanner doesn't expect.
withPlaceholders(sql) skips SELECT/FROM column parsing entirely and
only locates the ${where}, ${andWhere}, ${having} and ${andHaving}
placeholder positions, requiring explicit columnMapping() calls (as
with unparsed()). This lets dynamic where()/having() expressions be
injected into otherwise unparseable SQL.
Also fixes two bugs in the underlying placeholder-position splitting:
- static SQL following a ${having}/${andHaving} placeholder (e.g. a
trailing ORDER BY) was silently dropped when both a where and a
having placeholder were present
- using only ${having}/${andHaving} (no where placeholder) caused a
dynamically added HAVING clause to be appended after trailing
static SQL, producing invalid SQL
Changes:
- RawSqlBuilder.withPlaceholders(sql) + SpiRawSqlService.withPlaceholders()
- DRawSqlParser.parseAsTemplate() / parseTemplate() - placeholder-position
only parsing, correctly splitting preWhere/preHaving/trailing SQL
- CQueryBuilderRawSql - skip column-list and "select" prefix handling
in template mode (signalled by an empty preFrom)
- Unit tests in DRawSqlServiceTest covering where/andWhere/having/andHaving
placeholder combinations, including the two fixed edge cases
- Integration tests in TestRawSqlWithPlaceholders (ebean-test) covering
CTE queries with dynamic where/having and verifying generated SQL
* Add examples test using query beans
* Add docs guides for RawSql
* Add ${orderBy} ${andOrderBy}
---------
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
When only a @ManyToMany collection is modified and saved, the parent bean has no dirty scalar properties so BeanPersistController preUpdate/postUpdate were never invoked — only the junction table rows were written.
Fix mirrors the existing preElementCollectionUpdate() mechanism: add
preManyToManyUpdate() on PersistRequestBean and call it from
SaveManyBeans.saveAssocManyIntersection() when the intersection actually
changes (vanillaCollection, forcedUpdate, or a tracked BeanCollection
with non-empty additions or removals). The !dirty guard in
preManyToManyUpdate() prevents double-firing when the bean itself is
also dirty.
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
* Implement query.exists() via select exists(...)
Previously query.exists() was implemented by running a findIds() limited to 1 row and checking whether the list was empty. This PR replaces that with a dedicated select exists(select 1 ...) execution path.
## Changes:
• CQueryExists — new query executor, analogous to CQueryCount, that runs select exists(...) and returns the boolean result directly from the JDBC ResultSet
• CQueryBuilder.buildExistsQuery() — builds select exists(select 1 ...) SQL, reusing the query plan cache on repeat calls
• CQueryEngine.findExists() / DefaultOrmQueryEngine / OrmQueryEngine interface — wire the new executor through the standard query engine stack, including SQL logging, summary logging, and query cache put
• DefaultServer.exists() — delegate to request.findExists() instead of findIds()
## Tests added to TestQueryExists:
• testExistsBoolean_returnsFalse — verifies false is returned when no rows match
• testExistsBoolean_withJoin — exercises the join SQL path in buildExistsQuery; also checks false when the join yields no match
• testExistsBoolean_queryPlanReuse — runs the same query twice and asserts the generated SQL is identical, confirming the plan cache is hit on the second call
* Fix TestQueryExists only
---------
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
* Fix#3821 fetchQuery secondary tables not included in query cache dependent tables
When a query uses fetchQuery("path"), the secondary table(s) must be registered as dependent tables in the query cache entry so that updates to those tables properly invalidate the cache.
## Fix:
In CQueryPlan, after building the SQL-tree dependent tables, walk the query's OrmQueryDetail for any fetchQuery paths and resolve each path through the root BeanDescriptor to its target entity's baseTable(). These tables are merged into dependentTables at plan construction time (once, then cached in the plan).
This means the fix applies uniformly whether paths were specified via fetchQuery("path") calls or via query.select(fetchGroup) — both populate the same OrmQueryDetail.
### Tests added to TestQueryCacheTableDependency:
• fetchQuery_oneToMany_invalidatesQueryCacheOnSecondaryTableUpdate — updating a contact invalidates a Customer query cache that includes fetchQuery("contacts"); asserts the refreshed result contains the updated phone number
• fetchQuery_manyToOne_invalidatesQueryCacheOnSecondaryTableUpdate — updating a root record invalidates an ECacheChild query cache that includes fetchQuery("root")
* Ensure tests cleanup their test data
* query.secondaryQuery() (called during prepareQuery()) removes fetchQuery paths from OrmQueryDetail before CQueryPlan is built. The existing buildDependentTables iterated detail.entries() looking for isQueryFetch() entries — but they were already gone.
Changes:
1. OrmQueryRequest.java — added secondaryQueries() getter to expose the already-stored SpiQuerySecondary field.
2. CQueryPlan.java — rewrote buildDependentTables to accept SpiQuerySecondary instead of OrmQueryDetail. It now iterates secondary.queryJoins() (the paths already removed from detail) and adds each path's base table to the dependent tables set.
---------
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
* Fix asOf() queries bypassing bean cache isolation
asOf() queries must never read from or write to the bean cache since the cache holds only current state. Previously, lazy loads and secondary queries spawned by an asOf parent could return stale current data from the cache instead of the historical snapshot.
- DefaultServer.findId(): skip bean cache check when isAsOfQuery()
so the findById fast-path doesn't return current state for temporal queries
- DLoadContext: force useBeanCache=CacheMode.OFF when asOf != null so hitCache=false for all lazy load batch cache checks, and propagate CacheMode.OFF to secondary (+query) joins
- LoadBeanRequest.configureQuery(): when !loadCache (parent context disabled cache), explicitly set CacheMode.OFF on the lazy-load SQL query — previously left at CacheMode.AUTO which caused cache reads/writes even when the parent had cache disabled
Fixes#3713
* Simplify asOf to CacheMode.OFF
---------
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
In createJsonObjectMapperType(), remove the instanceof DeployBeanProperty
branch that overrode ownerType() with getField().getDeclaringClass().
For a generic superclass B<T>, getDeclaringClass() returns B with T
unresolved, so Jackson treated the field as Object and deserialized to
LinkedHashMap. DeployBeanProperty.ownerType() already returns
desc.getBeanType() (the concrete subclass), which gives Jackson the full
supertype context needed to resolve T correctly.
Co-authored-by: robin.bygrave <robin.bygrave@eroad.com>
Rob Bygrave