Automatically Denormalizing Document Relationships

Automatically Denormalizing
Document Relationships
Will Thompson
O'Connor's

(2) if there are sufficient reserves in the Railroad
Retirement Account, whether—
(A) the rates of such taxes should be reduced, or
(B) any part of the tax imposed by section 3221(b)
of title 26 should be diverted to the Railroad Unemploy-
ment Insurance Account to aid in the repayment of its
debt to the Railroad Retirement Account.
History of 45 U.S.C. §231f-1: Aug. 12, 1983, P.L. 98-76, §502, 97 Stat. 440;
Oct. 22, 1986, P.L. 99-514, §2, 100 Stat. 2095; Dec. 21, 1995, P.L. 104-66, §2221(a),
109 Stat. 733.
§231g [§8]. COURT JURISDICTION
Decisions of the Board determining the rights or li-
abilities of any person under this subchapter shall be
subject to judicial review in the same manner, subject
to the same limitations, and all provisions of law shall
apply in the same manner as though the decision were
a determination of corresponding rights or liabilities
under the Railroad Unemployment Insurance Act (45
U.S.C. 351 et seq.) except that the time within which
proceedings for the review of a decision with respect to
an annuity, supplemental annuity, or lump-sum benefit
may be commenced shall be one year after the decision
will have been entered upon the records of the Board
and communicated to the claimant.
History of 45 U.S.C. §231g: Aug. 29, 1935, ch. 812, §8, as restated June 24,
1937, ch. 382, 50 Stat. 307, as restated Oct. 16, 1974, P.L. 93-445, §101, 88 Stat.
1343.
See also 20 C.F.R. pt. 260.
ANNOTATIONS
Rivera v. U.S. R.R. Ret. Bd., 262 F.3d 1005, 1008
(9th Cir.2001). “[T]o qualify for review in this court
[under §231g], [claimant] must show that the [Rail-
road Retirement] Board’s dismissal of his claim consti-
tutes a ‘final decision of the Board.’” See also 45 U.S.C.
§355(f) (judicial-review provision of Railroad Unem-
ployment Insurance Act, incorporated into §231g).
Compare Abbruzzese v. U.S. R.R. Ret. Bd., 63 F.3d 972,
974 (10th Cir.1995) (without constitutional question
raised by refusal to reopen, courts of appeals lack juris-
diction to review Board’s decision not to reopen case),
with Sones v. U.S. R.R. Ret. Bd., 933 F.2d 636, 638
(8th Cir.1991) (Board’s decision not to reopen case is
reviewable under abuse-of-discretion standard).
nish employees with statements of their compensation
as reported to the Board. The Board’s record of the com-
pensation so returned shall be conclusive as to the
amount of compensation paid to an employee during
each period covered by the return, and the fact that the
Board’s records show that no return was made of the
compensation claimed to have been paid to an em-
ployee during a particular period shall be taken as con-
clusive that no compensation was paid to such em-
ployee during that period, unless the error in the
amount of compensation returned in the one case, or
the failure to make return of the compensation in the
other case, is called to the attention of the Board within
four years after the day on which return of the compen-
sation was required to be made.
History of 45 U.S.C. §231h: Aug. 29, 1935, ch. 812, §9, as restated June 24,
1937, ch. 382, 50 Stat. 307, as restated Oct. 16, 1974, P.L. 93-445, §101, 88 Stat.
1343.
See also 20 C.F.R. pt. 209.
ANNOTATIONS
Pawelczak v. U.S., 931 F.2d 108, 109 (D.C.Cir.
1991). “[A]s a matter of law, RRA §9 [now 45 U.S.C.
§231h] imposes the equivalent of a statute of limita-
tions. If the employee does not challenge the accuracy
of compensation records ‘within four years after the
day on which return of the compensation was required
to be made,’ the employee loses the opportunity to chal-
lenge those records. [¶] To facilitate employees’ com-
pliance with this requirement, the [Railroad Retire-
ment] Board’s regulations require railroad employers
to file a yearly compensation report for each employee
with the Board by February of the following year. The
Board, in turn, notifies the employee of the amount of
compensation the employee has reported. Under RRA
§9, the employee then has four years within which to
challenge the accuracy of the report.” See also Gate-
wood v. U.S. R.R. Ret. Bd., 88 F.3d 886, 889 (10th Cir.
1996).
§231i [§10]. ERRONEOUS PAYMENTS
(a) Recovery.—If the Board finds that at any time
more than the correct amount of annuities or other
benefits has been paid to any individual under this sub-
45 U.S.C. §§231f-1 - 231i
Chapter 9. Retirement of Railroad Employees
Railroad Retirement Act of 1974
!
45
U.S.C.
§231
f
-1
1. Deadline to file. A motion to transfer for improper venue is waived if it is made after any written mo-
tion (other than a special appearance) is filed. TRCP 86(1). The motion to transfer may be filed concurrently with
the answer. TRCP 86(2); see CPRC §15.063. See “Deadline to Answer,” ch. 3-E, §2, p. 255.
3-4. DEADLINES FOR MOTIONS TO TRANSFER VENUE
Grounds Deadlines Authority Cross-reference
1 Improper county and
convenience
Before or with filing of D’s
answer
CPRC §15.063(1); TRCP 86(1) §2.2.1, this page
2 Local prejudice None Common law §3.4, p. 237
3 Consent None CPRC §15.063(3); TRCP 86(1) §4.1, p. 238
2. Due order of pleading.
(1) Consent or improper county & convenience. The defendant must file a motion to transfer
venue based on consent or improper county and convenience before or along with all other pleadings or motions ex-
cept the special appearance, which must be filed first. TRCP 86(1). See “Due Order of Pleading,” ch. 3-A, §3, p. 207.
The defendant waives its objection to improper venue if it files a motion to transfer after it files an answer. See TRCP
86(1); Kshatrya v. Texas Workforce Comm’n, 97 S.W.3d 825, 832 (Tex.App.—Dallas 2003, no pet.).
NOTE
Although a motion to dismiss under TRCP 91a is not an exception to the due-order-of-pleading
rule, a defendant can file a motion to dismiss without waiving the motion to transfer venue.
TRCP 91a.8. See “No waiver of special appearance or motion to transfer venue,” ch. 3-H,
§2.7.1, p. 286.
(2) Local prejudice. The due-order-of-pleading rule does not apply to a motion to transfer based
on local prejudice under TRCP 257-259. See “Local Prejudice,” §3, p. 236.
3. Form. The motion to transfer venue must be in writing and may be made either as part of the defen-
dant’s first responsive pleading or as a separate document. TRCP 86(1), (2). See O’Connor’s Texas Forms, FORMS
3C:1-3.
4. No affidavits necessary. The defendant may, but is not required to, support the motion with affida-
vits when it is filed. TRCP 86(3) (last paragraph); GeoChem Tech v. Verseckes, 962 S.W.2d 541, 543 (Tex.1998).
The question of proper venue is raised by simply objecting to the plaintiff’s venue choice through a motion to trans-
fer venue. Billings v. Concordia Heritage Ass’n, 960 S.W.2d 688, 692 (Tex.App.—El Paso 1997, pet. denied). But
once the plaintiff responds to the motion and denies the defendant’s venue facts, the defendant must provide proof
as required by TRCP 87(3). See TRCP 87(2).
5. Request hearing. The defendant must request a hearing, give the plaintiff notice of the hearing, and
secure a setting for the hearing. See TRCP 87(1); see, e.g., Carlile v. RLS Legal Solutions, Inc., 138 S.W.3d 403, 408
(Tex.App.—Houston [14th Dist.] 2004, no pet.) (14-month delay between filing motion to transfer and securing hear-
ing showed lack of diligence); Bristol v. Placid Oil Co., 74 S.W.3d 156, 159 (Tex.App.—Amarillo 2002, no pet.)
(32-month delay between motion to transfer and ruling was not attributable to D because D’s motion asked court to
Commentaries
Chapter 3. Defendant’s Response & Pleadings
C. Motion to Transfer—Challenging Venue
!
Motion
to
Transfer
Venue
O'Connor's
Print Books

Modeling relationships
One-to-one
Clients
ID FirstName LastName LogInEmail
112345 Tim Malone tim@xyz.com
223456 Sally Mott sally@abc.org
<client id="112345" ﬁrstName="Tim"
lastName="Malone"
loginEmail="tim@xyz.com" />
<client id="223456" ﬁrstName="Sally"
lastName="Mott"
loginEmail="sally@abc.org" />

One-to-many
Clients
ID FirstName LastName LoginEmail
<client id="112345" ﬁrstName="Tim"
lastName="Malone"
loginEmail="tim@xyz.com">
<phone-number label="Home">202-555-1654</
phone-number>
<phone-number label="Mobile">202-555-1876</
phone-number>
</client>
<client id="223456" ﬁrstName="Sally"
lastName="Mott"
loginEmail="sally@abc.org">
<phone-number label="Work">408-555-2780</
phone-number>
</client>
Client-phones
Client_ID PhoneLabel PhoneNumber
112345 Home 202-555-1654
112345 Mobile 202-555-1876
223456 Work 408-555-2780

Many-to-many
Clients
ID FirstName LastName LoginEmail
Attorneys
ID FirstName LastName
498456 Erin Baily
171239 Alan Davis
Client-Attorney
Client_ID Attorney_ID
239918 498456
191340 171239

Many-to-many
<xml>
?

Many-to-many relationships
Alternative : Discard relationship data
<client id="112345"
firstName="Tim" lastName="Malone">
...
</client>
<client id="223456"
firstName="Sally" lastName="Mott">
...
</client>
<attorney id="498456"
firstName="Erin" lastName="Baily">
...
</attorney>
firstName="Alan" lastName="Davis">
...
</attorney>

ﬁrstName="Bob" lastName="Shapiro">
<client id="538989"
ﬁrstName="O.J." lastName="Simpson">
...
</client>
<client id="185703"></client>
</attorney>
Alternative : Degrade relationship

ﬁrstName="Johnnie" lastName="Cochran">
<client id="538989"
...
</client>
</attorney>
<client id="538989"
...
</client>
</attorney>

<client id="538989"
...
</client>
</attorney>
ﬁrstName="Johnnie" lastName="Cochran">
<client id="538989"
...
</client>
</attorney>

<attorney id="123">
...
<attorney-client idref="abc"/>
<attorney-client idref="xqy"/>
</attorney>
<client id="abc">
...
<client-attorney idref="123"/>
</client>
<client id="xyz">
...
</client>
Keys and Joins

• Join functions must be explicitly built
• Reads are multiplied
<attorney id="123">
...
</attorney>
//client[@id="abc"]
//client[@id="xyz"]
...
<attorney id="012">
<attorney id="234">
...
Keys and Joins : Views

Keys and Joins : Joining on related data
<attorney id="123">
<address>...<state>CA</state>...</address>
</attorney>
<client id="abc">
<address>...<state>ME</state>...</address>
</client>
"West Coast attorneys representing East Coast clients"

<attorney id="123">
</attorney>
<client id="abc">
</client>
let $states-west := (‘CA’, ‘OR’, ‘WA’)
let $states-east := (‘ME’, ‘NH’, ‘RI’, ... , ‘FL’)
let $clients-east := //client[address/state = $states-east]
let $attorneys-west := //attorney[address/state = $states-west]
return $attorneys-west
[attorney-client/@idref = $clients-east/@id]
• Big join
• May read large portions of the
database
• Hard to optimize reliably

<attorney id="123">
</attorney>
<client id="abc">
</client>
return $attorneys-west
• Big join
database

<attorney id="123">
</attorney>
<client id="abc">
</client>
return //attorney[address/state = $states-west]
• Big join
database

<attorney id="123">
</attorney>
<client id="abc">
</client>
let $clients-east-ids := //client[address/state = $states-east]/@id
return //attorney[address/state = $states-west]
[attorney-client/@idref = $clients-east-ids]
• Big join
database

• Facets are calculated for entire result set
• Indexing facilitates fast facet calculations on
values in result documents
• Joins required for related document values
• Query cost scales with size of result set

• Facets are calculated for entire result set
• Indexing facilitates fast facet calculations on
values in result documents
• Joins required for related document values
• Query cost scales with size of result set
let $client-results := db:implementation-deﬁned(...)
let $client-attorney-refs := $client-results/client-attorney/@idref
return subsequence(
for $attorney in //attorney[@id = $client-attorney-refs]
let $count := count($client-results
[client-attorney/@idref = $attorney/@id])
order by $count descending
return <attorney-facet
value="{ $attorney/full-name }"
count="{ $count }" />,
1, 5)

Keys and Joins : Code maintenance
Organization
• Two commingled document
concepts
• Fractured codebase
Abstraction
• Join-based model harder to
generalize
• Dependencies

Automatic denormalization
Overview
Precomputation
Shifts responsibility from run-time to write-time
Conceptually similar to SQL indexed/materialized views
Explicit trade-off
Run-time performance
Simplicity

Overview
<client id="112345"
...
<ref:copies>
<attorney-client idref="498456" />
...
</attorney>
</ref:copies>
</client>
...
<ref:copies>
<client-attorney idref="112345" />
...
</client>
<client-attorney idref="223456" />
...
</ref:copies>
</attorney>

<client id="112345"
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>
...
<ref:copies>
<client id="112345"
...
</client>
<client id="223456">
...
</ref:copies>
</attorney>
Overview

<client id="112345"
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>
...
<ref:copies>
<client id="112345"
...
</client>
...
</ref:copies>
</attorney>
Relationship create

Entity copy transformation
<client id="112345"
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>
<client id="112345"
...
</client>
...
<ref:copies>
<client id="112345"
...
</client>
</ref:copies>
</attorney>

Update
<client id="112345"
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>

Update
<client id="112345"
ﬁrstName="Timothy" lastName="Malone">
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>

Update
<client id="112345"
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>
<client id="112345"
...
</client>
Entity copy transformation

Update
<client id="112345"
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>
...
<ref:copies>
<client id="112345"
...
</client>
</ref:copies>
</attorney>
Get referenced document
<client id="112345"
...
</client>

Update
<client id="112345"
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>
...
<ref:copies>
<client id="112345"
...
</client>
</ref:copies>
</attorney>
<client id="112345"
...
</client>
Update referenced document with copy

Update
<client id="112345"
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>
...
<ref:copies>
<client id="112345"
...
</client>
</ref:copies>
</attorney>
<client id="112345"
...
</client>

Delete
<client id="112345"
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>

Delete
<client id="112345"
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>
...
<ref:copies>
<client id="112345"
...
</client>
</ref:copies>
</attorney>
Get referenced document

Delete
<client id="112345"
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>
...
<ref:copies>
<client id="112345"
...
</client>
</ref:copies>
</attorney>
Remove copy from referenced document

Delete
<client id="112345"
...
<ref:copies>
...
</attorney>
</ref:copies>
</client>
...
<ref:copies>
...
...
</client>
</ref:copies>
</attorney>

Read
<client id="112345"
<address> ... <state>CA</state> ... </address>
<ref:copies>
<address> ...
<state>NY</state> ...
</address>
...
</attorney>
</ref:copies>
...
</client>
<address> ... <state>NY</state> ... </address>
<ref:copies>
<client id="112345"
<address> ...
<state>CA</state> ...
</address>
</client>
...
</ref:copies>
</attorney>

Read
let $clients-east := //client[state = $states-east]
return
//attorney
[address/state = $states-west]
<ref:copies>
<client id="112345"
<address> ...
</address>
</client>
...
</ref:copies>
</attorney>

Read
return
//attorney
[ref:copies/client//state = $states-east)]
<ref:copies>
<client id="112345"
<address> ...
</address>
</client>
...
</ref:copies>
</attorney>

Read
return
//attorney
[ref:copies/client//state = $states-east]
<ref:copies>
<client id="112345"
<address> ...
</address>
</client>
...
</ref:copies>
</attorney>
• Single document-scoped query
• Join eliminated

Read
return
//attorney
[ref:copies/client//state = $states-east]
<ref:copies>
<client id="112345"
<address> ...
</address>
</client>
...
</ref:copies>
</attorney>
• Join eliminated
• Indexable!

Read
let $client-attorney-refs := $client-results/client-attorney/@idref
return subsequence(
for $attorney in //attorney[@id = $client-attorney-refs]
let $count := count($client-results
[client-attorney/@idref = $attorney/@id])
1, 5)

Read
return subsequence(
for $attorney-id in distinct-values($client-results//attorney/@id)
let $count := count($client-results[//attorney/@id= $attorney-id])
1, 5)
• Join eliminated
• Even more indexable....

Read
• Join eliminated
Get values directly from index!
return subsequence(
for $attorney-id in distinct-values($client-results//attorney/@id)
let $count := count($client-results[//attorney/@id= $attorney-id])
1, 5)

Read
let $facets := cts:values(
cts:path-reference("/ref:copies/attorney/full-name",
(), "limit=5", $client-query)
for $f in $facets
let $count := cts:frequency($f)
return <attorney-facet value="{ $f }" count="{ $count }" />
• Join eliminated
MarkLogic

Read
let $facets :=
util:index-keys($client-results/ref:attorney/full-name, (),
function($key, $count) {
<attorney-facet value="{$key}" count="{$count[2]}" />
}, 5, "lucene-index")
for $f in $facets
order by $f/@count
return $f
eXist-db
• Join eliminated

YOU'RE GONNA HAVE A BAD TIME
Caveats : Relationship-heavy workloads
and/or update-heavy workloads
High frequency updates
High number of relationships per document
Overlapping relationships
Additional write lock contention

Caveats : Only recommended for two-way joins
• Possible to support 3 (or more)-way joins
• Combinatorial explosions

Extensions and Optimizations :
Sparse entity copy transformation
• Extend transformation to exclude
parts of copy
• Simple to follow rules based on:
containing entity
copied entity
combination
• Simplify documents
• Decrease update commit overhead
declare function ref:make-copy(
$source as element(),
$target as element()
) as element()
{
typeswitch($source)
(: Rules based on copied entity :)
case element(attorney) | element(client) return
From: Attorney or Client + To:Anything rules
case element(matter) return
(: Combination rules :)
typeswitch($target)
case element(client) return From:Matter + To:Client rules
default return From:Matter + To:Not-Client rules
default return
typeswitch($target) return
(: Rules based on containing entity :)
}

Entities and sub-entities
• More idiomatic
• More flexible
• Propagate ancestor/descendant data via
entity copy transformation updates

Overlapping documents
• One tree modeled as "base" document
• Overlapping documents modeled using many-to-many relationship
• After denormalization, all documents are completely coherent

Nearline reference updates
Update queue
Commit canonical document update
Queue denormalized copy update
Allows dirty reads "Eventually consistent"
Tunable queue More control over resource utilization

Conclusions
XML databases are great if you need
XML (or JSON)
No database is a panacea
More complex models will require
trade-offs
Automatic denormalization patterns are a
good bet for many-to-many relationships
‣ Simple
‣ Eliminates runtime dependencies
‣ Faster to code, faster to query
Application relationship size and update
behavior can break it
Not in production...still testing No real-world data yet

Automatically Denormalizing Document Relationships

More Related Content

Similar to Automatically Denormalizing Document Relationships (20)

Recently uploaded (20)

Automatically Denormalizing Document Relationships