Feature/merge works - #985

Draft

olovy wants to merge 249 commits into

developfrom

feature/merge-works

Commits on Jun 27, 2023

Clean up cluster merging script

olovy
authored and
kwahlin
committed
9bc5ccf
Add instructions

olovy
authored and
kwahlin
committed
1ecd468
Clean up

olovy
authored and
kwahlin
committed
7d2f60f
Clean up

olovy
authored and
kwahlin
committed
e96f546
Add script for getting responsibilityStatement

olovy
authored and
kwahlin
committed
d0674bd
Add script for getting responsibilityStatement

olovy
authored and
kwahlin
committed
23054b1
Add script for getting responsibilityStatement

olovy
authored and
kwahlin
committed
b3bcbe5
Update instructions

olovy
authored and
kwahlin
committed
785d653
Skip generic titles e.g. "Poems"

olovy
authored and
kwahlin
committed
1cd1276
Remove 'VariantTitle', 'CoverTitle' from title comparison

olovy
authored and
kwahlin
committed
06ad9e2
Add more ignored subtitles

olovy
authored and
kwahlin
committed
3704de1
Check extent

olovy
authored and
kwahlin
committed
7f94f4f
Fix generic subtitle handling

olovy
authored and
kwahlin
committed
1be4bab
Add Extent TODO

olovy
authored and
kwahlin
committed
26bf359
Add naive contribution linking

olovy
authored and
kwahlin
committed
97e2cb8
Fix title cluster printing

olovy
authored and
kwahlin
committed
255c9d5
Fix title cluster printing

olovy
authored and
kwahlin
committed
0cf6199
Add naive contribution linking

olovy
authored and
kwahlin
committed
251706c
Add naive contribution linking

olovy
authored and
kwahlin
committed
e840b3d
Add naive contribution linking

olovy
authored and
kwahlin
committed
554c881
Fix fiction filter

olovy
authored and
kwahlin
committed
999b950
Naming

olovy
authored and
kwahlin
committed
447c571
Add more generic titles

olovy
authored and
kwahlin
committed
9af6957
Add more MARC fiction terms

olovy
authored and
kwahlin
committed
784ca67
Add naive contribution linking

olovy
authored and
kwahlin
committed
c855694
Fix typo

olovy
authored and
kwahlin
committed
5dc8c89
Add naive contribution linking

olovy
authored and
kwahlin
committed
cea11e4
Add naive contribution linking

olovy
authored and
kwahlin
committed
922fa43
Add naive contribution linking

olovy
authored and
kwahlin
committed
3a1806c
Add naive contribution linking

olovy
authored and
kwahlin
committed
1fed900
Add naive contribution linking

olovy
authored and
kwahlin
committed
5800fb1
Add naive contribution linking

olovy
authored and
kwahlin
committed
d1c535d
Add naive contribution linking

olovy
authored and
kwahlin
committed
fa23f10
Add naive contribution linking

olovy
authored and
kwahlin
committed
a17c60b
Add naive contribution linking

olovy
authored and
kwahlin
committed
ecb83c5
Add naive contribution linking

olovy
authored and
kwahlin
committed
4fbaed0
Add naive contribution linking

olovy
authored and
kwahlin
committed
080681f
Clean up work clustering script

olovy
authored and
kwahlin
committed
be56fc7
Update initial cluster search parameters

olovy
authored and
kwahlin
committed
d36545a
Merge summary

olovy
authored and
kwahlin
committed
ee68978
New method for picking final work title

olovy
authored and
kwahlin
committed
bffbac9
New method for picking final work title

olovy
authored and
kwahlin
committed
b116fdd
New method for picking final work title

olovy
authored and
kwahlin
committed
678d91c
Display work a bit nicer in show2

olovy
authored and
kwahlin
committed
e4dcefb
Add verbose mode to link-contribution

olovy
authored and
kwahlin
committed
b9a0013
Handle extra chars in lifeSpan check

olovy
authored and
kwahlin
committed
ed998a2
Fix linkContribution storeAtomicUpdate

olovy
authored and
kwahlin
committed
abfa245
Add more testes for extent

olovy
authored and
kwahlin
committed
d8db886
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
b249707
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
748073e
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
7e6d8bb
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
f7a335e
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
62a0c74
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
e6421d6
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
0c70a74
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
3cfeeed
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
ef816c1
Add script for moving summary supplied by some providers from work to instance

olovy
authored and
kwahlin
committed
51c0915
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
f894a6c
Add script for moving summary supplied by some providers from work to instance

olovy
authored and
kwahlin
committed
d490547
Add script for moving summary supplied by some providers from work to instance

olovy
authored and
kwahlin
committed
88eb8b5
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
487ae86
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
ac1475d
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
e72772e
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
591bada
Add script for moving Elib cover designers

olovy
authored and
kwahlin
committed
30e631e
Add more generic titles

olovy
authored and
kwahlin
committed
2267fa1
Let intendedAudience marc/Juvenile match empty

olovy
authored and
kwahlin
committed
0f2ff25
Add filter for translation without translator

olovy
authored and
kwahlin
committed
2a0d82f
Handle translationOf with different type

olovy
authored and
kwahlin
committed
d3d3bc7
Handle translationOf with different type

olovy
authored and
kwahlin
committed
e616c23
Link contribution: More aggressive normalization of name strings

olovy
authored and
kwahlin
committed
2fa6e7e
Move summary to instance before merging works

olovy
authored and
kwahlin
committed
850a4cd
Move summary to instance before merging works

olovy
authored and
kwahlin
committed
47efe14
No more drama

olovy
authored and
kwahlin
committed
1122166
Add placeholder technicalNote to extracted work

olovy
authored and
kwahlin
committed
b439e05
Also output clusters of size 1 when fitering

olovy
authored and
kwahlin
committed
b13f73f
Move HTML stuff to its own file

olovy
authored and
kwahlin
committed
d62f2b7
Print report when merging

olovy
authored and
kwahlin
committed
93837be
mkdir report dir

olovy
authored and
kwahlin
committed
b4c66a3
Print report when merging

olovy
authored and
kwahlin
committed
1ca8d97
Print report when merging

olovy
authored and
kwahlin
committed
720635d
Print report when merging

olovy
authored and
kwahlin
committed
1a3cbe4
Print report when merging

olovy
authored and
kwahlin
committed
253bb8c
Print report when merging

olovy
authored and
kwahlin
committed
aebb252
Print report when merging

olovy
authored and
kwahlin
committed
dd5c803
Fix store()

olovy
authored and
kwahlin
committed
201ab1e
Fix URI

olovy
authored and
kwahlin
committed
d823566
Fix URI

olovy
authored and
kwahlin
committed
0366e1f
Add script for removing language from title

olovy
authored and
kwahlin
committed
fba2b53
Add script for removing language from title

olovy
authored and
kwahlin
committed
982b58f
Add script for removing language from title

olovy
authored and
kwahlin
committed
ea4ad2d
Add script for removing language from title

olovy
authored and
kwahlin
committed
7c410ed
Add script for removing language from title

olovy
authored and
kwahlin
committed
6a36adb
Add script for removing language from title

olovy
authored and
kwahlin
committed
8d574d7
Add script for removing language from title

olovy
authored and
kwahlin
committed
4e61faf
Add script for removing language from title

olovy
authored and
kwahlin
committed
09c39ae
Check isTranslationWithoutTranslator

olovy
authored and
kwahlin
committed
e7e8c33
Check isTranslationWithoutTranslator

olovy
authored and
kwahlin
committed
8b1d5c5
Try new rule for anonymous translations

olovy
authored and
kwahlin
committed
d71d931
Don't check extent

olovy
authored and
kwahlin
committed
1ee5edb
Fix -s2

olovy
authored and
kwahlin
committed
68efdda
Add filter -qm

olovy
authored and
kwahlin
committed
d3d9f61
Add filter -qm

olovy
authored and
kwahlin
committed
bd02be4
Add script for moving up contributions from responsibilityStatement
kwahlin
committed
8877dcf
Don't lose summaries when merging works

olovy
authored and
kwahlin
committed
1988b46
Drop generic subtitles when picking extracted work title

olovy
authored and
kwahlin
committed
3cf7756
Drop generic subtitles when picking extracted work title

olovy
authored and
kwahlin
committed
3d0e5fd
Update ignored-subtitles.txt

olovy
authored and
kwahlin
committed
8a4c4e1
Make likelyRolePattern stricter and make roleSpecified stat clearer
kwahlin
committed
7a1679a
Make thread safe
kwahlin
committed
2a93628
Properly remove contributor after match
kwahlin
committed
52fda10
Add missing dependency

olovy
authored and
kwahlin
committed
6ce4913
remove language only from work titles
kwahlin
committed
3b095c5
filter out works having contribution in relationship
kwahlin
committed
a2c8021
Add revert method
kwahlin
committed
896f6ef
Feature/resp statement to contribution (#1118)
kwahlin
committed
8ce37f0
Enable filtering clusters by contribution role
kwahlin
committed
06ceba6
Add date catalogue to work report link
kwahlin
committed
8b1add4
Add [Publit] to move-summaries-to-instance script

olovy
authored and
kwahlin
committed
0a6635a
Show reproductionOf links within cluster
kwahlin
committed
66cc802
Use date as job identifier
kwahlin
committed
7dcc2eb
Change catalogue structure
kwahlin
committed
0a30419
Handle labels in list
kwahlin
committed
7b3ac57
Fix revert
kwahlin
committed
18c7eb9
Fix removal of works
kwahlin
committed
5034adf
Handle exception when failing to find doc to remove
kwahlin
committed
fea0050
Get encodingLevel properly
kwahlin
committed
7dbf9c4
Save updates in right place
kwahlin
committed
8fc45b1
Don't reset changed flag to false
kwahlin
committed
be0c1e7
Prefix numPages
kwahlin
committed
09d019d
Fix genreForm merging
kwahlin
committed
4bcf364
Try new method for picking best work title
kwahlin
committed
70e6cb5
Add grep for safety
kwahlin
committed
e20d332
Fix argument types
kwahlin
committed
cadd656
Set generationDate/generationProcess before saving any updates
kwahlin
committed
2e8dedd
Drop all subtitles
kwahlin
committed
cddf246
Exclude local subject entities from merge unless type is ComplexSubject
kwahlin
committed
b477d39
Exclude local genreForm entities from merge
kwahlin
committed
c9b5fdd
Add showHubs option
kwahlin
committed
72f62e5
Fix title overview
kwahlin
committed
3d7c5c4
Only show hubs/clusters with more than 1 member
kwahlin
committed
7e27730
Improve view for unstored extracted works while getting rid of the Doc2 class
kwahlin
committed
2054cf8
Use Doc instead of Doc2, filter on derivedFrom
kwahlin
committed
dbdc53b
Add command line option for adding 9pu codes to illustrators
kwahlin
committed
c9f5cb1
Fix view for titles
kwahlin
committed
36dae38
Filter out fields with null values properly
kwahlin
committed
92bb0e0
Actually copy work
kwahlin
committed
00ec6d4
Print html only if cluster generates new merged work
kwahlin
committed
1ed9517
Don't parse editors for now
kwahlin
committed
0196509
Pile ids vertically in _derivedFrom
kwahlin
committed
73e2aa0
Don't add title source until after partition and prefer work title when biggest partitions are of equal size
kwahlin
committed
09e2624
Put scripts in separate files to run normally with whelktool
kwahlin
committed
bb5d956
WIP: refactor: separate Doc business and display logic

olovy
authored and
kwahlin
committed
02e080c
Clean up

olovy
authored and
kwahlin
committed
17501e7
Move out unneeded stuff from Util
kwahlin
committed
01963b8
Minor fixes
kwahlin
committed
5157aac
Adjust for when there will be linked works already
kwahlin
committed
0d103fd
Add new files too...
kwahlin
committed
0d7d354
Don't report if not stored
kwahlin
committed
7addd79
Split instead of tokenize
kwahlin
committed
5e8473d
Add missing size()
kwahlin
committed
a84e109
Remove exit method
kwahlin
committed
6960b38
handle titles in translationOf
kwahlin
committed
7202c02
Remove unneeded method
kwahlin
committed
9212e28
Fix reproductionOf display
kwahlin
committed
5397fd7
Add draft for scripted job and update some whelktool scripts accordingly
kwahlin
committed
9f6c1e2
Change order
kwahlin
committed
b55c2a5
Go back to prioritizing work titles over instance titles and drop genereic subtitles
kwahlin
committed
f0dd340
Add flag option for report dir
kwahlin
committed
31be1d2
Take into account that translationOf can be list when comparing
kwahlin
committed
18009af
Add missing parameter for storeAtomicUpdate
kwahlin
committed
ce4f020
Add num-threads flag
kwahlin
committed
a976b27
Remove return statement that prevents instances from being linked to their work
kwahlin
committed
6975a70
Add missing parameter for storeAtomicUpdate again
kwahlin
committed
af342a9
Fix report uri in technicalNote
kwahlin
committed
c2ec727
Copy executor service from Whelktool
kwahlin
committed
d0208ef
Make possible to pass Whelktool flags to wrapper script and print what's going on in each step
kwahlin
committed
e8a906e
Add script for adding missing translationOf to translations
kwahlin
committed
26be9ae
Better naming
kwahlin
committed
df50014
Prepare scripts to be run regularly
kwahlin
committed
7e89e44
Set generationProcess and generationDate for new works too
kwahlin
committed
88b8f74
Don't filter out anonymous translations and include translationOf normalization in wrapper script
kwahlin
committed
23c6a66
Always produce report with split clusters
kwahlin
committed
5725555
More concise where clause
kwahlin
committed
c79cba7
Enable cluster reporting with already merged works
kwahlin
committed
613b557
Reverse direction when comparing encoding levels since higher index means lower level
kwahlin
committed
5aa753d
Skip adapter -> editor normalization for now
kwahlin
committed
60f6977
Add generic subtitle 'berättelser för barn'
kwahlin
committed
5956a39
Keep blank gf terms / subjects
kwahlin
committed
6680c15
Add apostrophe-like symbol to noise list
kwahlin
committed
f05bb6c
Match untitled translationOf with those that have a title and pick best title when merging
kwahlin
committed
339d9f3
Exclude tactile text works from selection
kwahlin
committed
58fe196
Merge SAB classification only when codes are equal
kwahlin
committed
e1c04a8
Fix classification display
kwahlin
committed
7b5f03a
Add missing parenthesis
kwahlin
committed
8b0b4ef
Redirect annoying Whelktool output
kwahlin
committed
0ac3a71
Interlink cluster members with closeMatch
kwahlin
committed
547b0ed
Use right method for loading docs when collecting title clusters
kwahlin
committed
99f7fda
Ignore too large result sets in find-work-clusters.groovy

olovy
authored and
kwahlin
committed
7c5c564
Remove ES query operators from e.g. titles in find-work-clusters

olovy
authored and
kwahlin
committed
5916f7f
Don't do fuzzy title search in find-work-clusters

olovy
authored and
kwahlin
committed
87d06df
Refine closeMatch linkage and start general restructuring (WIP)
kwahlin
committed
ed6d139
Add urval to generic titles
kwahlin
committed
a30001c
WIP: More refactoring
kwahlin
committed
3ef5f8c
Make report methods work after refactoring
kwahlin
committed
6ecffec
Remove unused methods
kwahlin
committed
b44be93
Don't save unmodified documents
kwahlin
committed
470b686
Allow copying contribution from records with lower encoding level
kwahlin
committed
3613e0b
Match only titles of type Title, not ParallelTitle
kwahlin
committed
13e7b8b
Ignore more subtitles
kwahlin
committed
9b842e4
Add more selection criteria (LXL-4147)
kwahlin
committed
8101f9f
Append partNumber/partName to mainTitle when copying title from instance to work (LXL-4149)
kwahlin
committed
972f255
Link instances to work, not to self
kwahlin
committed
7dbbecb
More refactoring (towards eliminating Datatool)
kwahlin
committed
6c8097c
Add ʼ character to noise list
kwahlin
committed
90ef6ec
Output only clusters with at least two records when filtering Swedish fiction
kwahlin
committed
f3f88ce
Bin some redundant classifications when merging
kwahlin
committed
e65665d
Trim whitespace from SAB codes
kwahlin
committed
0e25cfd
Filter out records whose titles have more than one part from selection
kwahlin
committed
4c23451
Filter out manuscripts from selection
kwahlin
committed
238eece
Drop generic subtitles appearing as substrings
kwahlin
committed
9a242da
Make saogf/L%C3%A4ttl%C3%A4st distinguishing
kwahlin
committed
75520fd
Look for saogf/Handskrifter too when deciding if manuscript
kwahlin
committed
1f61554
Create all necessary directories before writing reports
kwahlin
committed
85756cb
Make sure incompatible works never end up in same group when partitioning
kwahlin
committed
60627cc
WIP: Add new script for *all* contribution related normalizations on work clusters
kwahlin
committed
77743b3
Implement more normalization methods and extensive reporting
kwahlin
committed
2d05453
Add missing translationOf if translator in contribution
kwahlin
committed
251721c
Remove replaced scripts
kwahlin
committed
11e868a
Be more restrictive about adding local entities in contribution (avoid duplicates)
kwahlin
committed
8e3dae2
Finalize script
kwahlin
committed
39845cc
Add script for moving some illustrators in work clusters to instance
kwahlin
committed
96b1d99
Look for more gf terms when deciding if illustrator should be moved
kwahlin
committed
2921926
Move more roles to instance
kwahlin
committed
4fded0a
Include contributions to instance in wrapper script
kwahlin
committed
aa64202
Add script for specifying designer roels in elib records (and move unspecified to instance)
kwahlin
committed
1c8f416
Move more roles to instance by looking at domain
kwahlin
committed
9668408
Report all moved roles
kwahlin
committed
29c74e9
Map Formgivare to bookDesigner instead of designer
kwahlin
committed
fd1094f
Fix report dir typo
kwahlin
committed
151ccec
Fix role link
kwahlin
committed
535c196
Remove print
kwahlin
committed
72147e8

Commits on Jun 28, 2023

Uh oh!

Commits

Commits on Jun 27, 2023