Newspack Opens Up Application Process for Phase Two

Earlier this year, Newspack chose twelve publications to take part in the initial rollout phase of the platform. Newspack is a collection of themes, plugins, and features geared towards newsrooms such as revenue generation wizards, mobile delivery, and search engine optimization.

Steve Beatty, head of Newspack Communication says they’re seeking up to 50 newsrooms to be part of phase two which lasts from September 1st – February 29th, 2020.

“What you’ll get: a new Newspack website, including the migration of your existing site; free hosting, security, updates, backups and support on WordPress.com through February 2020; membership in the Newspack community of users; access to Newspack developers; exclusive performance benchmarking against your peers; and more,” Beatty said.

Organizations that are selected are expected to provide feedback, test new features, and help shape the overall direction of the platform.

Free hosting for charter members will expire on February 29th, 2020. News organizations with revenue under $500K can expect to pay $1,000 per month and organizations that generate revenue of over $500K will pay $2,000 per month. Newspack is currently in negotiations to provide subsidies for organizations that encounter difficulties with the pricing structure.

Those interested in participating in the charter program have until August 15th to fill out the application.

Google Launches Effort to Make Robots Exclusion Protocol an Internet Standard, Open Sources Robots.txt Parser

Website owners have been excluding web crawlers using the Robots Exclusion Protocol (REP) on robots.txt files for 25 years. More than 500 million websites use robots.txt files to talk to bots, according to Google’s data. Up until now, there has never been an official Internet standard, no documented specification for writing the rules correctly according to the protocol. Over the years, developers shared their various interpretations of the protocol, but this created many different ambiguous methods for controlling crawlers.

Google is working together with Martijn Koster, the original author of the protocol, webmasters, and other search engines to create a proposal to submit to the Internet Engineering Task Force (IETF) for standardizing the REP:

The proposed REP draft reflects over 20 years of real world experience of relying on robots.txt rules, used both by Googlebot and other major crawlers, as well as about half a billion websites that rely on REP. These fine grained controls give the publisher the power to decide what they’d like to be crawled on their site and potentially shown to interested users. It doesn’t change the rules created in 1994, but rather defines essentially all undefined scenarios for robots.txt parsing and matching, and extends it for the modern web.

The proposed specification includes several major items that webmasters and developers will want to review. It extends the use of robots.txt to any URI-based transfer protocol (FTP, CoAP, et al), instead of limiting it to HTTP. It also implements a new maximum caching time of 24 hours and lets website owners update robots.txt whenever they choose, without having crawlers overload their sites with requests. If a previously accessible robots.txt file becomes inaccessible for whatever reason, crawlers will respect the known disallowed pages that were previously identified for “a reasonably long period of time.”

Google has also open sourced the C++ library it uses for parsing and matching rules in robots.txt files, along with a testing tool for testing the rules. Developers can use this parser to create parsers that use the proposed REP requirements. It has been updated to ensure that Googlebot only crawls what it’s allowed to and is now available on GitHub.

“This library has been around for 20 years and it contains pieces of code that were written in the 90’s,” Google’s Search Open Sourcing team said in the announcement. “Since then, the library evolved; we learned a lot about how webmasters write robots.txt files and corner cases that we had to cover for, and added what we learned over the years also to the internet draft when it made sense.”

Lizzi Harvey, who maintains Google’s Search developer docs, updated the robots.txt spec to match the REP draft. Check out the full list of changes if you want to compare your robots.txt file to the proposed spec. If the proposal for standardizing the REP is successfully adopted by the IETF, the days of googling and wading through undocumented robots.txt rules will soon be over.

CMS Backend Opener: A Firefox Extension to Quickly Locate the Login Page to Popular CMS Backends

If you use Firefox and manage multiple websites that use different Content Management Systems and have a hard time keeping track of the various URLs to their backends, consider using the CMS Backend Opener Firefox extension created by Andy R.

Once installed, you can use either a keyboard shortcut (Alt + Y) or press a button within the browser and it will automatically open the login page for the detected CMS in a new window.

The extension uses the CMS meta-tag: Generator to detect which CMS is being used. The following CMS’ are supported:

  • Typo3
  • Typo3 Neos
  • Joomla
  • WordPress
  • Django
  • Shopware (beta)
  • Magento (beta)
  • Drupal
  • Contao
  • Weblication
  • WebsiteBaker
  • CMSQLite
  • Oxid

Although the extension has not been updated in two years, I tested it on Firefox 67.0.4 on my MacBook Pro and it worked without any issues. I typically use a bookmark to browse to WP-Admin but this is more convenient, especially on WordPress.com.

I’ve also learned that if you have Pretty Permalinks enabled in WordPress, you can type /login or /admin after your domain and it will typically load the login page.

Lessons from the GraphQL Documentary: Never Underestimate the Power of Open Source Communities

Honeypot, a tech-focused job platform based in Europe, has produced a documentary that offers a fascinating look at the origins of GraphQL. The 28-minute video explores how quickly the project began to have an impact on the wider tech industry after Facebook publicly released it as an open source project.

GraphQL co-founder Nick Schrock, who was interviewed along with fellow co-creators Lee Byron and Dan Schafer, said the documentary “captured both the urgency and joy of the early months of the GraphQL.” It was filmed over two months in San Francisco and Berlin, where Honeypot runs the GraphQL Conf in cooperation with Prisma.

GraphQL began as an internal project at Facebook that was born out of necessity as the tech industry began to shift towards providing better mobile experiences for users. At that time, Facebook’s native apps were just a thin wrapper around the mobile website.

“The inability of a large technology company to adjust to a technology shift as big as the mobile shift is the type of thing that will consign a seemingly unstoppable empire to the grave in a matter of a few years,” Schrock said.

Facebook decided to re-write the Facebook iOS app but the APIs they had at that time were inadequate for creating the Newsfeed. A new Newsfeed API was written simultaneously to be used with the new mobile app. Facebook for iOS 5.0, released in 2012, was a native re-write of the app and also the first time GraphQL was deployed in the wild. Following that release, its use was expanded beyond just the Newsfeed to encompass most of the functionality offered in Facebook’s iOS app.

Facebook shared GraphQL with the world at React Europe 2015 and published the GraphQL spec later in 2015. They explained that their goal was to design what they thought was the ideal API for frontend developers and work backwards with the technology.

GraphQL’s creators were surprised at how fast the uptake was after making the project public. Engineers at Airbnb, Twitter, and Github were early adopters and their experiences are shared in the documentary with interviews from the community. The problems GraphQL’s creators encountered in scaling their mobile experience were not specific to Facebook. Other companies had similar problems and the demand for GraphQL in the industry was already there. Within six months, the team saw implementations of GraphQL in many of the major programming languages. They realized how important the project was to the industry after GitHub announced in 2016 that its public API would be a GraphQL API:

Using GraphQL on the frontend and backend eliminates the gap between what we release and what you can consume. We really look forward to making more of these simultaneous releases. GraphQL represents a massive leap forward for API development. Type safety, introspection, generated documentation, and predictable responses benefit both the maintainers and consumers of our platform.

The documentary tells the story of how GraphQL began the first three years as a solution to internal problems at Facebook but expanded to become a community tool that was initially adopted by hobbyists and then incorporated into the products of large tech companies. GraphQL co-founder Lee Byron predicts that the project is entering the next phase of its life and “heading towards becoming an industry standard and one that’s collaborative.”

There’s no way to measure the number of APIs that are being built around GraphQL, but the query language is now used in both internal and external APIs at major companies like Pinterest, Intuit, Coursera, Walmart, Shopify, PayPal, KLM, NBC News Digital, Credit Karma, Wayfair, and Yelp. Since it can be used in combination with REST APIs, GraphQL’s rapid adoption is not necessarily a good predictor for the end of REST architecture, but it’s a trend that is worth following. This widespread adoption began with just a handful of engineers who saw GraphQL’s promise at React Europe 2015, built tools to optimize development, and advocated for using GraphQL at their companies.

“I totally underestimated the power of these open source communities,” Schrock said. “We had to rely on this community of poeple to form spontaneously and then build implementations of this in different languages and then actually productionize it and build an entire tool ecosystem around it. I didn’t think that was ever going to work, and I was totally wrong. If an idea makes sense to people and it clicks with their mind and they can see the vision, they are actually willing to do a lot of work in order to see it executed and share their work, and it’s a pretty remarkable thing to see.”

The energy in the GraphQL documentary is inspiring and the story shares many parallels with other open source projects that have gained widespread adoption through passionate communities. Check out the full documentary below: