Schuman on GPO’s New and Improved U.S. Statutes at Large, with Bulk Metadata in MODS

Daniel Schuman, Esq., of the Sunlight Foundation, has posted GPO is Closing Gap on Public Access to Law at JCP’s Direction, But Much Work Remains, on the Sunlight Foundation Blog.

The post describes the U.S. Government Printing Office’s making available United States Statutes at Large, 1951-2009, with full text for individual statutes in PDF and .txt formats, and for entire volumes in PDF format, and metadata for individual statutes and in bulk for each volume in MODS format.

For example, click here for bulk metadata for volume 123 of U.S. Statutes at Large, in MODS format.

Here are excerpts from the post:

The GPO’s recent electronic publication of all legislation enacted by Congress from 1951-2009 is noteworthy for several reasons. It makes available nearly 40 years of lawmaking that wasn’t previously available online from any official source, narrowing part of a much larger information gap. It meets one of three long-standing directives from Congress’s Joint Committee on Printing regarding public access to important legislative information. And it has published the information in a way that provides a platform for third-party providers to cleverly make use of the information. While more work is still needed to make important legislative information available to the public, this online release is a useful step in the right direction. [...]

In mid-January 2013, GPO published approximately 32,000 individual documents, along with descriptive metadata, including all bills enacted into law, joint concurrent resolutions that passed both chambers of Congress, and presidential proclamations from 1951-2009. The documents have traditionally been published in print in volumes known as the “Statutes at Large,” which commonly contain all the materials issued during a calendar year. [...]‘

The online release of the Statutes at Large from 1951 forward, accompanied by good quality metadata, has made it possible for the public to see important (and not so important) legislation. [...] Of course, the major New Deal legislation, such as the Securities and Exchange Act of 1934, and anything from the last quarter of the 19th century, is still unavailable from GPO. And GPO/LOC still hasn’t incorporated the information it has released onto THOMAS. However, the free legislative information portal GovTrack already jumped in with both feet, incorporating this information on its website where users can easily search and find historical bills going back to 1951.

Depending on the quality of the searchable PDFs GPO has posted, it may also be just a matter of time before someone pull out the text from the PDFs and puts it into a database, too. That would improve the ability to search for bills and facilitate analysis of congressional activities. It also moves us one step closer to a very difficult but important goal: allowing people to see in real time how draft legislation would amend the law. For that to be possible, GPO would need to publish the Statutes at Large from prior to 1951 online in electronic form, including making clear the underlying structure of the statutes, plus there’d have to be some pretty significant advances in the tools available to parse legislative language. [...]

For more details, please see the complete post.

U.S. GPO releases House bills in bulk XML

Today the U.S. Government Printing Office has begun making U.S. House of Representatives’ legislative bills available for bulk download in XML, at http://www.gpo.gov/fdsys/bulkdata/BILLS

Currently, only bills from the current Congress are included.

The user guide provides more details about the data.

Click here for the GPO press release.

Click here for the Speaker’s office press release.

Release of the bills in bulk XML is a key goal of the #freeTHOMAS movement and a longstanding goal of the open government data community.

The House bills are the second major Congressional data set to be released in bulk XML this month; the first was the House floor proceedings in bulk XML.

Here are links to selected posts about the availability of this new legislative data set:

• Daniel Schuman: Access to Legislation Gets Better, Promise of More to Come
• Joshua Tauberer: On the new bulk bill XML from GPO
• Nick Judd: House Republicans Release More Data Catnip for Developers
• Alex Howard: Open data of U.S. House legislation now available in bulk format

Please see the comments to this post for links to additional resources about this issue.

Mill: U.S. House of Representatives floor proceedings in bulk XML

Eric Mill of the Sunlight Foundation points us to a new legislative bulk data resource: U.S. House of Representatives’ floor proceedings in bulk XML, 2005 to present (also called House Floor Activities Download).

The announcement of the availability of these data, by the Speaker of the House, states that this is the first project of the new Legislative Branch Bulk Data Task Force.

Bruce, Mill, Schuman, Tauberer & Wonderlich: Recommendations on Public Access to Legislative Data

Tom Bruce of the Legal Information Institute; Daniel Schuman, Eric Mill, and John Wonderlich, all of the Sunlight Foundation; and Dr. Joshua Tauberer of GovTrack and POPVOX, have posted a new report entitled On Public Access to Legislative Information: Recommendations to the Bulk Data Task Force (2012).

The report “provides a roadmap” that the U.S. Congress’s Bulk Data Task Force can use “to implement[]” free public “bulk access to” the THOMAS database of U.S. federal legislative information.

The report is a product of the effort — known as #freeTHOMAS — to provide free online public access in bulk to THOMAS.

For more information, please see Daniel’s post entitled How to #FreeTHOMAS: A report on implementing bulk access.

HT @danielschuman

Google Endorses freeTHOMAS: Free Public Access to U.S. Federal Legislative Data in Bulk XML

Google has announced its support for Congress’s providing free public access in bulk XML to U.S. federal legislative data, in a new post by Seth Webb, Google’s Senior Policy Manager, entitled In Support of Legislative Transparency, on the Google Public Policy Blog.

The effort to persuade Congress to provide free public access to U.S. federal legislative data in bulk XML is popularly known as freeTHOMAS (hashtag #freeTHOMAS).

Mr. Webb writes:

We believe the ability to download bulk legislative data in formats like XML on a regular basis provides tremendous benefits. Website and app developers can use such data to provide up-to-date information on bills. Researchers can use it to perform studies. And politically-curious citizens can use it to follow legislation moving its way through Congress. [...] We applaud Congress for the work that it’s done to promote openness and look forward to a future of increased legislative transparency.

For more information, please see the complete post.

For more information about freeTHOMAS, please see Schuman: Major Transparency Milestone in Bulk Access Statement; the new post by David Moore of Participatory Politics Foundation entitled Next Steps in #freeTHOMAS Campaign; and Open Congress’s THOMAS bulk data access wiki.

HT @garvinfo

Tauberer, Bruce, and Schuman Quoted: Congressional Data May Soon Be Easier to Use Online

David A. Fahrenthold has published Congressional Data May Soon Be Easier to Use Online, Washington Post, June 8, 2012.

The article describes recent efforts to petition Congress to make U.S. federal legislative data — specifically the data underlying the THOMAS legislative information system — publicly available free of charge in bulk XML. Those efforts are popularly known as the freeTHOMAS movement (hashtag #freeTHOMAS).

The article contains quotations from Dr. Joshua Tauberer of Govtrack, Tom Bruce of the Legal Information Institute, and Daniel Schuman of the Sunlight Foundation.

For more information about freeTHOMAS, please see Schuman: Major Transparency Milestone in Bulk Access Statement; the new post by David Moore of Participatory Politics Foundation entitled Next Steps in #freeTHOMAS Campaign; and Open Congress’s THOMAS bulk data access wiki.

Schuman: Major Transparency Milestone in Bulk Access Statement

Daniel Schuman of the Sunlight Foundation has posted Major Transparency Milestone in Bulk Access Statement, on the Sunlight Foundation Blog.

In this post, Mr. Schuman reports that on 6 June 2012 the leadership of the U.S. House of Representatives issued a statement expressing its support for providing public bulk access to U.S. federal legislative data, the goal of the movement commonly known as “freeTHOMAS” (Twitter hashtag #freeTHOMAS.)

In the statement the House leadership identified as one of its goals the “provi[sion of] bulk access to legislative information to the American people without further delay,” characterized provision of bulk access to legislative data as “among our top priorities in the 112th Congress,” described a new task force whose charge is “to expedite the process of making public information available to the public,” and “direct[ed] the task force to begin its [...] work immediately.”

The full text of the House leadership’s statement appears at the bottom of this post, and is also available here.

Mr. Schuman characterizes the House leadership’s statement as momentous, and expresses substantial optimism about the statement. He writes:

[T]oday is a milestone for legislative transparency. [...] The debate over whether there should be bulk access to legislative data is over. Because bulk access is a top priority of the 112th Congress, we expect to see tangible progress in the upcoming months. The remaining questions largely concern how bulk access should be implemented to meet the needs of the public while respecting the legitimate concerns of Congress and its support agencies.

Certain aspects of the House leadership’s statement raise questions, however. Mr. Schuman notes that the task force will include no individuals from outside of government, and that the task force does not appear to be required to consider input from the public. In addition, the House leadership’s statement does not set a deadline for the completion of the task force’s work, does not require the task force to issue a report or to make any of its deliverables public, and does not require the task force to make use of the 2008 Library of Congress memorandum that addresses many of the issues that appear to be on the task force’s agenda. Further, the language of the House leadership’s statement, to the extent that it is contained in a press release, appears to lack the authority of language in legislation or a congressional report or print.

Mr. Schuman expresses optimism that the task force will invite input from the public, and states that “Sunlight [Foundation] and our friends in the transparency community stand ready to be of assistance as the technical, policy, and scope issues are addressed.”

Mr. Schuman also suggests that the public should continue to monitor this issue, to make sure that the promise of the House leadership’s statement is fulfilled. He writes:

While this is clearly progress, there’s still much more to do. We will be monitoring this issue closely.

For more information, please see Mr. Schuman’s complete post.

Click here for other commentary about #freeTHOMAS.

Here is the full text of the House leadership’s statement:

House Leaders Back Bulk Access to Legislative Information

Posted by Speaker Boehner Press Office
June 6, 2012
Press Release

WASHINGTON, DC – House Speaker John Boehner (R-OH), Majority Leader Eric Cantor (R-VA), Legislative Appropriations Subcommittee Chairman Ander Crenshaw (R-FL), and Oversight & Government Reform Committee Chairman Darrell Issa (R-CA) released the following statement today regarding House efforts to provide bulk access to legislative information:

“The coming vote on the Legislative Branch appropriations bill marks an important milestone for the House of Representatives: the moment lawmakers agree to free legislative information from the technical limits of years past and embrace a more open, more transparent, and more effective way of doing the people’s business. Our goal is to provide bulk access to legislative information to the American people without further delay.

“The bill directs a task force to expedite the process of making public information available to the public. In addition to legislative branch agencies such as the Library of Congress and the Government Printing Office, the task force will include representatives of House leadership and key committees, as well as the Clerk of the House and the House Chief Administrative Officer.

“This is a big project. That’s why accomplishing it rapidly and responsibly requires all those with a role in the collection and dissemination of legislative information to be at the table together. Because this effort ranks among our top priorities in the 112th Congress, we will not wait for enactment of a Legislative Branch appropriations bill but will instead direct the task force to begin its important work immediately.

“The offices involved in this project have been instrumental in using new technology to make the House more open. We pledged to make Congress more transparent and accessible, and from our efforts to provide legislation and updates in XML, to the video streaming and archiving of committee hearings, to our search for new ways to engage and serve the American people through events like last year’s ‘Hackathon’ – and more – we’re working to keep that pledge. Bulk data is the next and a very important step. We look forward to the task force’s report and to beginning implementation of this project as soon as possible.”

# # # # #

House Committee Votes to Indefinitely Delay Public Bulk Access to U.S. Federal Legislative Data

[UPDATE, 7 June 2012]: Schuman: Major Transparency Milestone in Bulk Access Statement

[UPDATE, 5 June 2012, 18:00 Eastern: Daniel Schuman reports: Issa Offers #FreeTHOMAS Amendment to Leg Approps Bill.]

[UPDATE, 5 June 2012: Daniel Schuman says the language of the print has been amended; see his new post entitled Bulk Access Language Tweaked by Approps.]

The U.S. House Committee on Appropriations has voted to approve a committee print directing an indefinite delay on a decision whether to provide U.S. federal legislative data in bulk XML to the public, according to Daniel Schuman’s new post entitled Bulk Access Developments after the H. Approps Hearing, on the Sunlight Foundation Blog.

According to the approved committee print, the decision will be delayed indefinitely while a new task force meets to consider the committee’s concerns. Those concerns mostly involve what are described as the “unresolved” “challenge of authenticating downloads of bulk data legislative data files in XML,” and costs and other issues that purportedly may arise from the alteration of legislative data by downstream users.

Here is the language that, according to Mr. Schuman, has been approved by the committee:

During the hearings this year, the Committee heard testimony on the dissemination of congressional information products in Extensible Markup Language (XML) format. XML permits data to be reused and repurposed not only for print output but for conversion into ebooks, mobile web applications, and other forms of content delivery including data mashups and other analytical tools. The Committee has heard requests for the increased dissemination of congressional information via bulk data download from non-governmental groups supporting openness and transparency in the legislative process. While sharing these goals, the Committee is also concerned that Congress maintains the ability to ensure that its legislative data files remain intact and a trusted source once they are removed from the Government’s domain to private sites.

The GPO currently ensures the authenticity of the congressional information it disseminates to the public through its Federal Digital System and the Library Congress’s THOMAS system by the use of digital signature technology applied to the Portable Document Format (PDF) version of the document, which matches the printed document. The use of this technology attests that the digital version of the document has not been altered since it was authenticated and disseminated by GPO. At this time, only PDF files can be digitally signed in native format for authentication purposes. There currently is no comparable technology for the application and verification of digital signatures on XML documents. While the GPO currently provides bulk data access to information products of the Office of the Federal Register, the limitations on the authenticity and integrity of those data files are clearly spelled out in the user guide that accompanies those files on GPO’s Federal Digital System.

The GPO and Congress are moving toward the use of XML as the data standard for legislative information. The House and Senate are creating bills in XML format and are moving toward creating other congressional documents in XML for input to the GPO. At this point, however, the challenge of authenticating downloads of bulk data legislative data files in XML remains unresolved, and there continues to be a range of associated questions and issues: Which Legislative Branch agency would be the provider of bulk data downloads of legislative information in XML, and how would this service be authorized. How would ‘‘House’’ information be differentiated from ‘‘Senate’’ information for the purposes of bulk data downloads in XML? What would be the impact of bulk downloads of legislative data in XML on the timeliness and authoritativeness of congressional information? What would be the estimated timeline for the development of a system of authentication for bulk data downloads of legislative information in XML? What are the projected budgetary impacts of system development and implementation, including potential costs for support that may be required by third party users of legislative bulk data sets in XML, as well as any indirect costs, such as potential requirements for Congress to confirm or invalidate third party analyses of legislative data based on bulk downloads in XML? Are there other data models or alternative[sic] that can enhance congressional openness and transparency without relying on bulk data downloads in XML?

Accordingly, and before any bulk data downloads of legislative information are authorized, the Committee directs the establishment of a task force composed of staff representatives of the Library of Congress, the Congressional Research Service, the Clerk of the House, the Government Printing Office, and such other congressional offices as may be necessary, to examine these and any additional issues it considers relevant and to report back to the Committee on Appropriations of the House and Senate.

As Daniel Schuman and Eric Mill have noted (here and here), the committee print language respecting the task force is problematic from the perspective of those who favor increased government transparency, increased participation of the public in government policy, or efficiency in government operations, because, according to the language of the committee print, the task force:

• will have no members from outside of the government
• has no deadline to complete its work
• is not required to receive input from the public
• is not required to make its report public
• is not required to take into consideration the 2008 Library of Congress memorandum that already addressed this issue in detail.

For commentary about this new House policy, click here.

House Committee Considers Indefinitely Delaying Public Bulk Access to U.S. Federal Legislative Data

[NOTE, 4 June 2012: Daniel Schuman has just posted an update on this issue: #FreeTHOMAS.]

[NOTE, 1 June 2012: Daniel Schuman has just posted an update on this issue: Bulk Access Developments after the H. Approps Hearing.]

A draft U.S. congressional committee print released today calls for indefinitely delaying a decision whether to provide public access to U.S. federal legislative data in bulk XML.

In their new post entitled Appropriators May Undercut Legislative Transparency, Daniel Schuman and Eric Mill of the Sunlight Foundation describe the draft print and outline the concerns it raises respecting public access to legislative data.

A large number of posts about this issue are linked from the Open Congress page, THOMAS bulk data access. [HT Daniel Schuman]

Many citizens’ comments about the issue are posted here at POPVOX (scroll down). HT Joshua Tauberer.

David Moore comments on the print in his new post entitled Email Congress to Oppose H.R. 5882 and #FreeTHOMAS.

Dr. Joshua Tauberer comments on the print in his new post entitled Rep. Crenshaw thinks American public can’t be trusted with overseeing Congress.

Tom Bruce comments on the print in his new post entitled Can we stop talking about accountability for a minute? Please?

Waldo Jaquith also comments on the draft print in his new post entitled Congress Declines to Let People Download Copies of Bills.

David Moore of Open Congress has added comments about the draft print to his recent post entitled Congress Refuses to #FreeTHOMAS.

Sarah Lai Stirland comments on the draft print in her TechPresident post entitled Transparency Advocates Frustrated With House Appropriators’ Plan To Make A Plan.

Nick Judd reports on the draft print in his TechPresident post entitled For Transparency Advocates, the Honeymoon with House Republicans May Be Over.

The full U.S. House Appropriations Committee is scheduled to vote on the draft print tomorrow, 31 May 2012. If the committee votes to approve the draft print in its current form, the decision whether to provide bulk XML access to the data underlying the THOMAS congressional information system will be indefinitely delayed, while a task force addresses the committee’s concerns.

Those concerns mostly involve what are described as the “unresolved” “challenge of authenticating downloads of bulk data legislative data files in XML,” and costs and other issues that purportedly may arise from the alteration of legislative data by downstream users.

Advocates of greater public access to government information may wish to contact Representative Ander Crenshaw (202-225-2501), author of the draft print, to express their views respecting whether U.S. federal legislative data ought to be available to the public in bulk XML, as U.S. federal regulatory data already is (see Federal Register in bulk XML here, and Code of Federal Regulations in bulk XML here), and respecting the composition, charge, schedule, transparency, and public access to the deliverables of the proposed task force on public access to legislative data in bulk XML. Consider using these talking points written by Daniel Schuman.

For more information, please see Daniel Schuman and Eric Mill’s post, Waldo Jaquith’s post, and David Moore’s post.

Schuman: Two Steps Forward on Improving Public Access to Legislative Information

Daniel Schuman of Sunlight Foundation has posted Two Steps Forward on Improving Public Access to Legislative Information, on the Sunlight Foundation Blog.

In this post, Mr. Schuman discusses two pieces of evidence suggesting that Congress may eventually provide free public access in bulk to congressional legislative data in XML — specifically, the data underlying the THOMAS legislative information system.

The first piece of evidence is a comment from a member of Congress that such bulk access will be discussed at length in a forthcoming report from the House Appropriations Committee Legislative Branch Subcommittee.

The second is a newly released, 2008 memorandum from the Library of Congress Congressional Relations Office, describing a plan “to make the underlying raw THOMAS data available to the public in XML.”

Mr. Schuman comments:

What’s notable is how the Library of Congress was technologically positioned to deliver on legislative data transparency four years ago, but apparently did not move forward. At a minimum, it should alleviate concerns about the difficulty of technological implementation.

Mr. Schuman further notes that the memo discusses the question of “who owns the data” in THOMAS. He writes:

The memo raises one major policy implication concerning who owns the data, contemplating that it belongs to the House, Senate, Congressional Research Service, and Government Printing Office. In the literal sense, that’s backwards: the information is owned by the American people and held in trust by Congress and its legislative agencies. These entities do serve as repositories of the information, however, and deserve consideration as to the technological means by which it is made available. However, that’s with the understanding that these entities should strive to meet the public’s need for the information and expansively follow the policies set by Congress in favor of transparency.

For more information, please see the complete post.