DL:DISC status reports
The following status reports were copied on 10/2/13 from the DISC wiki:
Progress August 14-September 10 The DISC ISG had four meetings during this reporting period.
- Loading of FSU's second batch of DigiTool materials has begun. FSU now has materials from all four of the initial Content Models in Islandora: Basic Image, Large Image, PDF and Books.
- FAU's test site was set up: fau7t.fcla.edu, and sample PRISM and Yiddish Children's Books collections were loaded, along with a sample Book from the Florida Heritage collection. All of those materials are PALMM collections and also contain palm7t.fcla.edu collection codes and are viewable from that site.
- A PALMM test site has been set up, along with themed prototypes based on palmm.fcla.edu collections have been created, and the PRISM, Yiddish Children's Books, and Florida Heritage collections have been populated with materials loaded into the FSU test site, fsu7t.fcla.edu, and the FAU test site, fau7t.fcla.edu. The Florida Heritage collection contains both FSU and FAU materials.
- Discussions of the Islandora PALMM site are underway with ISG in preparation for DISC discussions.
- The Islandora unattended batch user interface is in production and contains data from FSU's production DigiTool migrations.
- FLVC has been working with FSU to finalize additional development requests for DGI.
- The unattended batch ingest program now has the ability to load ETDs with embargoes, and testing with FAU's DigiTool ETDs will begin shortly.
Progress July 10-August 13 Islandora:
- The first batch of FSU materials, representing approximately 35 collections, have been loaded into FSU's production site and reviewed and approved by FSU. At the same time these objects were also assigned PALMM collection codes so that the display in the PALMM production site currently under construction.
- FSU is undergoing Islandora training and workflow testing in preparation for going into production.
- FSU/FLCA implementation meetings.
- The conversion of FAU DigiTool objects has begun with the first two collections: PRISM, and Yiddish Children's Books. The FAU test and production sites are under construction.
- A prototype of the Islandora unattended batch user interface has been created and will be demonstrated at the next ISG call.
- Future Islandora development by DGI.
Progress June 12 - July 9 The DISC ISG had four meetings during this reporting period.
Current status: The display and searching features on the FSU test site (https://fsu7t.fcla.edu) are substantially complete, and the current code has been copied to the FSU site on the Islandora Production server. Initial FSU staff accounts have been set up on FSU's production site, and the site has been turned over to FSU to begin the creation of their Islandora collection objects. A larger test load on the Production server will begin shortly, and a smaller test load is currently underway on the FSU test site. In the meantime, new permissions code and embargo code have been delivered by DGI, and testing of the new code is underway on the Islandora development server. The new code will need to be integrated into the production server before FSU goes into production with Islandora.
Progress May 8 - June 11 The DISC ISG had four meetings during this reporting period.
Much work was done on the FSU test site to prototype display and searching, and that work is nearing completion. FSU test site: https://fsu7t.fcla.edu
FLVC work included:
- Contracting for extra development in the area of permissions and item embargos. Work is currently underway by DGI
- Decisions regarding Islandora site architecture that will allow all site-owned materials to be loaded into an institution's Islandora site, with PALMM being a display-only site and all item maintenance being performed on institution's sites.
- Work on PURLs for Islandora objects.
- Discussion of future development priorities.
- Multiple test loads of FSU DigiTool materials.
- Planning for FSU to go into production with Islandora. The exact date has not yet been set.
A PASIG webinar on 6/11 re: Islandora was presented and FLVC staff and some ISG members attended:
You can access the recording at http://www.asis.org/Conferences/webinars/Webinar-PASIG-6-11-2013-r9nB7.wmv
Progress April 10 - May 7
The DISC ISG had four meetings during this reporting period. Islandora v.7 code was installed on the Islandora Test server this month and the server was released to ISG for testing on April 16. The test server now has the following features:
- A prototype of site, collection, and object branding, with clickable object logo/wordmarks.
- Search faceting
- Simple and Full MODS metadata forms
- An example of the Islandora Books module with IA book reader
- A "view all objects" option to view/browse the entire repository
- Solr indexing of MODS metadata, including a prototype of collection-specific searching
A prototype of a structured Table of Contents display was implemented on the FLVC Islandora development server and development for the display is substantially completed. Development of the DigiTool to Islandora conversion programming is underway and will produce data in JSON format to utilize the structured Table of Contents display.
In addition, an FSU test site was set up to continue to work with display/branding customization and for preliminary testing and review of DigiTool conversions. Samples of Basic Image, Large Image, and PDF content types are being tested with the DigiTool to Islandora conversion programming. FSU has agreed to prototype additional display refinements on their Islandora test site.
FLVC and the ISG are reviewing functional requirements and documenting details of required functionality in preparation for additional development work.
Progress March 13 - April 9
The DISC ISG had four meetings during this reporting period. During March the Islandora version 7 production code was released. FLVC installed the code on our Islandora development server and tested a number of priority features and modules, including:
- The custom PDF development that allows ingest of existing OCR full-text.
- Drupal permissions features
- The new Books Module
In addition to testing new version 7 code, FLVC staff continued work on Solr indexing, focusing specifically on collection-specific searching. A variety of issues reported by ISG during Islandora Testing Cycle 1 were also addressed on the development server. Examples of Books Content Model were loaded on the development server and discussions of detailed requirements for Books Table of Contents were discussed and some preliminary ideas were mocked up and demonstrated.
Islandora version 7 production code was then installed on the Islandora test server in preparation for ISG Testing Cycle 2, and that cycle is expected to begin within the week.
The ISG worked on compiling a list of software requirements for Day 1 of Islandora production and a list of development priorities was created that will be revisited as necessary. Individual requirements are being defined in detail.
Progress February to date
Summary: To date FLVC has not received v.7 Islandora production code but we've been working with code for release candidates in order to continue to make progress.
DISC ISG had six meetings during the period of February - March 12. We worked on:
- Finalizing SUL disk requirements for Islandora through June 30, 2014.
- Finalized the requirements for the Full MODS edit form
- Testing of Basic Image, Large Image, single PDF content models and MARCXML support on the Islandora test server and recording and discussing issues.
- Began a review of the Ranked features survey to prioritize features and future development.
- Reviewed Basic Image, Large Image, and single PDF Content Model requirements in preparation for final testing and sign-off
- Viewed a mock-up of branding options by site, collection, and object.
FLVC staff worked on:
- Installation and review of Islandora v.7 release candidates on the Islandora development server and feedback to DGI.
- Work has begun on collection-specific permissions based on available v.7 pre-release code.
- A lot of progress has been made on Solr indexing and faceting, and exploration of collection-specific searching and faceting has begun.
- Installation and review of custom PDF Full Text code.
- Updated and posted requirements for Basic Image, Large Image, and single PDF Content Models. See https://islandora.pubwiki.fcla.edu/wiki/index.php/Main_Page#Content_Models
- Exploring branding options and creation of a mock-up of a branded site, collection, and objects for demonstration to DISC
- An inventory of DigiTool materials for migration to Islandora
- A project update was presented by Lydia Motyka to the FLVC Library Services Members Council on February 28.
- An Islandora Milestones historic overview was presented by Bill Covey to CSUL on March 8.
Progress through end of January 2013
DISC ISG had one meeting in January 2013. We worked on:
- Regrouping after staff changes at the end of December 2012
- Updating the Islandora development plan ()
- Finalizing the Full MODS edit form - not yet complete in January
- Developing an Islandora testing framework
- Making the ISG meetings
FLVC staff worked on:
- SOLR upgrade and improved full text indexing
- MODS metadata mapping (MODS-to-DC, MARC-to-MODS, etc)
- Testing export of materials from DigiTool to Islandora by Content Model
- Testing offline ingest of materials into Islandora
- Investigating disk requirements for future Islandora use
- Partnering with DGI for improved/customized PDF full text indexing
- Creating migration plans and inventories for DigiTool to Islandora materials
Progress through end of December 2012
DISC ISG had two meetings in December. We worked on:
- PDF display - The PDF file itself now displays embedded in the Summary tab. The group decided on the format of the Full Metadata tab as well, preferring rows of data.
- Large Image display - the preview of the Large Image display looked clean, simple, and intuitive.
- Search - the group decided on some of the functionality for Basic Search.
- Finalized the Simple MODS metadata entry form.
- Worked on mapping of MODS to OAI-DC
Since the last DISC call we've had 2 meetings of the ISG. The group finished several go-rounds on the unattended batch spec and we are programming to the approved spec. The group also finalized review of the Simple MOD Metadata Entry form and that is now available for use. The group reviewed the PDF content model and is requiring some changes that we have discussed with DGI. We had two discussions of display for PDFs, with no firm conclusion, and that will be revisited in future.
Islandora7t is now available for testing by ISG members. There are 4 logins available with different levels of privileges from submitter to collection manager. Only the Basic Image Content Model is working the way it should be, but Large Image and Single PDF are in the pipeline. Also, search, indexing and display of metadata are still not working off MODS but will be very soon.
The Form Builder for making new metadata entry forms was buggy when I reported last month but a newer version is working well. DGI just released a bunch of new code affecting nearly all of the Islandora modules. We are in the process of installing them, resolving config problems, and seeing how they work. Then we need to reapply our local changes, and push the code out to 7t so you can see it too. The new code includes the ability to import, download and view MARCXML. It also includes the new Large Image viewer which is really nice. I think there are more goodies but haven't seen it all yet.
We have a DGI call on Thursday and I will ask for more information about Newspapers.
We heard from Broward College that they are hoping to use our Islandora when it is up and running. He asked where to find a timeline or public information about the project, which made me realize we don't have public information available. I want to add this to FCLAWeb.
Since the last DISC call we have held 4 meetings of the ISG. One of the main things we did was discuss how unattended batch processing should work. FLVC presented a proposal which the group reviewed and made suggestions, resulting in a revised proposal which the group reviewed again and mostly accepted. A programmer is working on unattended batch now.
Another thing the ISG did was give requirements for the browse display and the display of search results. Search results is coded and the programmer is working on browse now. Search and browse are currently based on Dublin Core metadata and will have to be reviewed once MODS support is in place.
DGI is now working on coding for MARC support. They decided to first develop a framework for adding information (e.g. pre-existing metadata records) to Ingest, then use that framework for adding MARC metadata. Coding for the framework should be complete and we should be able to see this in our call on Thursday 11/15. Coding for MARC specifically will follow.
The PDF module has been ported to version 7 and will be the next thing we work on. DGI missed their October deadline for Large Image but claim they will have it by the end of November.
We continue to test and find problems with the Form Builder that prevent us from making MODS the primary metadata record.
Since the last DISC call there has only been one meeting of the Islandora SubGroup. (Two were scheduled, but Priscilla had to cancel one because of a conflict with FLVC meetings.) At the meeting, the Subgroup approved a mock-up of Object Display and Search Results Display, clarified a permissions issue, and discussed behaviors of the Basic Image Content Model. The next call is scheduled for October 11 at 11:00.
We had three more meetings with DGI. We now have a reviewed and agreed-upon method of handling basic permissions and the representation of owning and submitting institutions. The requirements document for Authorization is at https://docs.google.com/document/d/16HL1ZRaiWVDufl2Ka4puF405yzyKuOe9GG2J2EcHD9o/edit?usp=sharing.
Programming to implement authorization is mostly completed. Programming to handle the Basic Image display is done; the display will be reviewed by the Islandora Subgroup tomorrow. Programming to change the Basic Image Content Model to handle our behavior requirements is in process.
We have created 2 instances of Islandora 7: islandora7d.fcla.edu for the development team, and islandora7t.fcla.edu for the ISG. Drupal sites palmm7d.fcla.edu, ncf7d.fcla.edu and fau7d.fcla.edu have been created on the development instance and the equivalent sites will later be created on test.
We have had 3 meetings of the Islandora Subgroup since the last DISC call. The group -- decided which historical items to carry over from DigiTool (e.g. DigiTool pid, ingest_id, creation_date, etc.) and how to represent them in Islandora (those with other legacy systems should be thinking about how to treat their information) -- approved a simple metadata entry form -- reviewed and revised requirements for single image Content Models (Basic Image and Large Image) -- advised on the design of Object Display and Search Results Display. Caitlin and Favenzio are now working on implementing Object and Search Results displays.
Work continues with DGI on how to treat owning and submitting institutions for authorization, statistics and branding. DGI's first programming assignment will be a module adding MARC support. MARC records will not be stored in the system, but when creating new objects it will be possible to submit MARCXML metadata which will be converted by program to MODS; it will also be possible to display any MODS metadata as MARCXML.
A brief project plan was submitted to CSUL for their September meeting: https://docs.google.com/document/d/1fCfK5NUlv11BVxvO3py9NOaSXdL77_uisLLTs5RMIUk/edit?usp=sharing
Kickoff meeting with DGI took place. They urged us to revise our plan and implement on Islandora 7 (the Drupal 7 version) which is out in beta (barely) right now. This was a major blow to the project plan, which we need to completely revise. We will lose about 3 months while we wait for all Islandora 6 modules to be ported to 7.
We sent our first requirements document to DGI. It covers the way we want our implementation to work for us as a consortium building some collections collectively and others individually. This will affect metadata, permissions, and branding.
Our development Islandora system was upgraded from 6.12.1 to 6.12.2. The new version contains a new Book Solution Pack that uses the Internet Archive BookReader, which is much better. These changes will be applied soon to the Test system, Islandorat. We also installed a development instance of Islandora 7. We can use it to learn the new architecture, but there is not much working besides the Basic Image Solution Pack.
The Islandora Subgroup met twice more. The group decided that a single data entry form be used for all types of materials, and that it be very simple. When editing, however, all possible fields will be available. The group also reviewed the architectural plan and suggested that PALMM be a discovery layer on top of all the institutional collections. We're looking into the implications of that.
Randy and Caitlin went to Islandora Camp on Prince Edward Island. Randy is back, Caitlin went on to SAA.
A local Islandora test machine (islandorat.fcla.edu) was opened up to DISC Islandora Subgroup members for their use. There are four userids set up with different permissions. Islandorat is one version behind the latest released version, and is scheduled to be upgraded by the end of July.
The Islandora Subgroup met twice more. Questions posed to the group and decisions made can be viewed at https://docs.google.com/document/d/1I6OlaFZBfo-hOYhRnqBekLyZxo9pOSXHwEg2M9Q-__8/edit?usp=sharing.
The DigiTool PALMM and institutional collections have been reviewed and Content Models for most of the content have been provisionally ascertained. Questions will be taken back to the content owners. Libraries have been updating the descriptions of their non-DigiTool content and we are starting to review those. Libraries who have not provided this information have been reminded to provide it.
A formal request has been made to FLVC leadership to fill Astrid's vacant position, which is a librarian in Digital Library Services who would largely be dedicated to the Islandora effort. (Note that the Gainesville Office of the FLVC, formerly known as FCLA, has no authority to hire staff.)
Our "kickoff" call with Discovery Garden Inc (DGI) will be scheduled for the week of July 16th. Until we review the DGI assignments and timing, we can not finalize the project plan. Islandora Subgroup members have been added to the Zoho project management system so they can access the draft plan.
We have two Islandora machines up and running with all available Solution Packs installed. One is an internal development machine, the other is a VM that we will open up to the DISC Islandora Subgroup members by the end of the month.
We have signed a contract with Discovery Garden Inc (DGI) for 255 hours of consulting and development services to be completed by December 15. FSU is contributing funds to the project that will allow us to contract for additional work in FY12/13. We will be putting together a plan of work for DGI that does not overlap development being done at other sites. For work in progress elsewhere, see: https://docs.google.com/document/d/1acIFQjb1RQY1tMsAsIBLFpuEr0QwMQMvqI66vwD7vbE/edit?usp=sharing
The Islandora Subgroup has met twice and a third call is being scheduled. Questions posed to the group and decisions made can be viewed at https://docs.google.com/document/d/1I6OlaFZBfo-hOYhRnqBekLyZxo9pOSXHwEg2M9Q-__8/edit?usp=sharing.
An overview project plan and timeline is due to CSUL at the end of June, 2012. We will share it with DISC for comment before sending.
Participation needed by other parties:
- We need all libraries with non-DigiTool collections to complete a collection description template. So far only FIU has done this work.
- Existing collections should use the template https://docs.google.com/spreadsheets/d/1n0lyq_AZIiRi0HMZvvQTUvLA1_AIdkpnSKbiwUmz_o0/edit?usp=sharing. **Prospective collections use the template https://docs.google.com/spreadsheets/d/1j6jQ9ZdjXz_2BxVU9Odw2rLNRNkZ9-YX2m0f6WAsWro/edit?usp=sharing.
- Review and comment on draft specification for Newspaper Solution Pack. Mark Sullivan and Emily Gore have access to this document.
We continue to work on installing local versions of Islandora. This is not trivial because there are many components, including (but not limited to) Fedora, GSearch, SOLR, Drupal, Islandora, Islandora Solution Packs and several databases. We have one system mostly built on a development machine and our sysadmins are in the process of building a test system on a VM. Both should be finished by the end of this week. In the meantime we are testing with the "canned" VM downloaded from the Islandora website, and the "sandbox" that they provide on the website.
We are re-evaluating the Islandora system against the checklist of requirements. (In process, see https://docs.google.com/document/d/1QmsctPhRDXSKvpCu525MFh7zfuD5GORfQlzdpMNCLuU/edit?usp=sharing) This is part of the process of trying to scope the size of the implementation project and to prioritize implementation and development tasks. On Monday 5/7 Emily Gore (FSU), Mark Sullivan (UF), and FCLA staff met to review the situation so far. We are in the process of assembling:
- a list of issues to take to DISC or a working group
- a list of items requiring configuration or development
The plan for this month is to:
- complete installation of the development and test machines
- begin working with DISC on the issues list
- contact Islandora sites that are known to be developing or contracting for development and see what we can get from them. These include UCLA, CARL members, and the University of Pittsburgh.
- contact Discovery Garden to ascertain their availability, cost, terms, and find out what they are working on for other sites.
- finish identifying and prioritizing development and configuration items
- begin negotiation with Discovery Garden for development
The following Solution Packs are available for installation:
- Basic Image https://wiki.duraspace.org/display/ISLANDORA113/basic+image+solution+pack
- Large Image https://wiki.duraspace.org/display/ISLANDORA113/large+image+solution+pack
- Book https://wiki.duraspace.org/display/ISLANDORA113/books+solution+pack
- PDF https://wiki.duraspace.org/display/ISLANDORA113/PDF+Solution+Pack
- audio https://wiki.duraspace.org/display/ISLANDORA113/audio+solution+pack
- video https://wiki.duraspace.org/display/ISLANDORA6121/Video+Solution+Pack
These Solution Packs are in development:
- document by UPEI (no ETA)
- Institutional Repository by UPEI (no ETA)
- ETD by the University of Northern British Columbia (no ETA)
- enhanced Book (by University of Pittsburgh, no ETA)
These materials are supported in other Islandora systems but not as Solution Packs:
- Oral history (by Colorado College)
FCLA has started working on the Islandora project. It has been slow getting off the ground because of limited staff time and a steep learning curve.
- Islandora test system(s). The canned Islandora VM that was downloaded for the Common Platform Review Group is available for limited testing. We are also installing the system on a development machine, the first step in creating our own local test instance of Islandora. The operating system components and Fedora are installed, next need SOLR, Drupal, and Islandora modules.
- Content modeling. Fedora requires content models for all data streams that it manages. Content models specify for a given object type, what kind of files will be included and what kind of behaviors they will have. Getting the content models right is the most important part of a Fedora implementation. We have been reading through the documentation on content models and "playing with" them in our local Fedora instance. We are also in the process of building a spreadsheet identifying the various types of objects in DigiTool (PALMM and institutional collections). See https://docs.google.com/spreadsheets/d/1n0lyq_AZIiRi0HMZvvQTUvLA1_AIdkpnSKbiwUmz_o0/edit?usp=sharing
- Solution packs. Islandora includes some content models in its "Solution Packs" which bundle everything you need for various types of content (e.g. books, large (jp2) images, etc.). We need to determine whether existing Islandora Solution Packs are adequate for our own use, and what new and modified Solution Packs are in the pipeline. This has taken a back-burner to content modeling but will be a focus in April.