Documentation PR’s Welcome - Why Docs Are Not A Beginner Friendly Task

Recently I was reporting a usability issue with a library, mainly related to it’s confusing or absent documentation. A friend of mine saw the exchange and commented (quite accurately) that it went along the lines of:

  • Me: This library should improve it’s documentation
  • Project: PR’s welcome
  • Me: I can’t write the docs because I don’t know how this works without documentation

My friend also commented “[this] is probably the realest interaction I’ve seen in a while.” and “It kinda shakes up the idea that if people want something in OSS they should do it themselves, and that docs are the easiest way to contribute”.

I completely agree with these observations.

Documentation writing is not a task for a drive by contributor, or a beginner. Documentation writing is a skill in and of itself, that requires many things. From my perspective, I believe documentation writing requires:

  • Linguistic ability - to express ideas clearly to a diverse audience of readers.
  • Emotional intelligence - to empathise with their audience and to think “what would help them in these words”?
  • Technical knowledge - the understanding of the problem space to know how to write the correct documentation.

Generally, when someone is a beginner to a project they lack an understanding of the project audience so they aren’t able to empathise with “what could be useful for a reader”. They also tend to lack the depth and breath of technical knowledge to know what to write for the documentation since by definition, they are a newcommer to this project.

A great way to connect with a beginner is to listen to their frustrations and what challenges the encountered with your project. First, connect how that frustration could be either a failing of the user interface and it’s design (even an API is a user interface). Second, only once you have eliminated design limitations, then consider it to be a documentation problem. As the project contributor, you are in a better position to write documentation than a beginner.

Telling someone, especially a beginner, “docs PR’s welcome” is both belitting to the skill that is technical writing, but also generally considered in opensource to mean “I don’t give a shit about your problem, fuck off”.

How CTAP2.0 made UserVerification even more confusing

I have previously written about how Webauthn introduces a false sense of security with how it manages UserVerification (UV) by default. To summarise, when you request “preferred” which means “perform UV if possible”, it can be bypassed since relying parties’s (RP) do not check if UV was actually performed, and Webauthn makes no recommendations on how to store credentials in a manner that allows future checking to ensure UV is requested or validated correctly.

From this, in Webauthn-RS we made the recommendation that you use either “required” to enforce all credentials have performed UV, or “discouraged” to request that no UV is performed by credentials during authentication or registration.

At the same time, in the Webauthn-RS project we begun to store two important pieces of credential metadata beyond the Webauthn specification - the result of UV from registration, and the policy that was requested at the time of registration. We did this because we had noticed there were classes of credentials, that even in “discouraged” would always verify themself at registration and authentication. Because of this property, we would enforce that since UV was performed at registration, we could continue to enforce UV on a per credential basis to detect possible credential compromise, and to further strengthen the security of credentials used with Webauthn-RS.

This created 3 workflows:

  • Required - At registration and authentication UV is always required
  • Discouraged + no UV - At registration and authentication UV is never required
  • Discouraged + always UV - At registration and authentication UV is always required

A bug report …

We recieved a bug that an authenticator was failing to work with Webauthn-RS, because at registration it would always force UV, but during authentication it would never request UV. This was triggering our inconsistent credential detection, indicating the credential was possibly compromised.

In this situation, the authenticator used an open-source firmware, so I was able to look at the source and identify the programming issue. During registration UV is always required, but during “discouraged” in authentication it’s never required matching the reported bug.

The author of the library then directed me to the fact that in CTAP2.0 this behaviour is enshrined in the specification.

Why is this behaviour bad?

I performed a quick poll on twitter, and asked about 5 non-technical friends about this. The question I asked was:

You go to a website, and you’re asked to setup a yubikey. When you register the key you’re asked for a pin. Do you now expect the pin to be required when you authenticate to that website with that yubikey now?

From the 31 votes on twitter, the result was 60% (21 / 30) that “yes” this PIN will always be required. From the people I asked directly, they all responded “yes”. (This is in no way an official survey or significant numbers, but it’s an initial indication)

Humans expect things to behave in a consistent manner. When you take an action one time, something will always continue to behave in that way. The issue we are presented with in this situation is that CTAP2.0 fundamentally breaks this association by changing the behaviour between registration and authentication. It also is not communicated that the different is registration vs authentication, or even why this behaviour is changed.

As a result, this confuses users (“Why is my pin not always required?!”) and this can at worst cause users to be apathetic about the UV check, where it could be downgraded from “required/preferred” to “discouraged” and the user would not notice or care about “why is this different?”. Because RP’s that strictly follow the Webauthn specification are open to UV bypass, CTAP2.0 in this case has helped to open the door for users to be tricked into this.

The other issue is that for a library like Webauthn-RS we lose the ability to detect credential compromise or attempts to bypass UV when in discouraged, since now UV is not consistently enforced across all classes of authenticators.

Can it be fixed?

No. There are changes in CTAP2.1 that can set the token to be “always verified” and for extensions to be sent that always enforce UV of that credential, but none of these assist the CTAP2.0 case where none of these elements exist.

As an RP library author we have to assume and work out ways to interact with credentials that are CTAP2.0_pre, CTAP2.0, CTAP2.1, vendor developed and more. We have to find a way to use the elements at hand to create a consistent user interface, that also embed security elements that can not be bypassed or downgraded.

I spent a lot of time thinking about how to resolve this, but I can only conclude that CTAP2.0 has made “discouraged” less meaningful by adding in this confusing behaviour.

Is this the end of the world?

Not at all. But it does undermine users trust in the systems we are building, where people may end up believing that UV is pointless and never checked. There are a lot of smart bad-people out there and they may utilise this in attacks (especially when combined with the fact that RP’s who strictly follow the Webauthn standard are already open to UV bypass in many cases).

If the goal we have is to move to a passwordless world, we need people to trust their devices behave in a manner that is predictable and that they understand. By making UV sometimes there, sometimes not, it will be a much higher barrier to convince people they can trust these devices as a self contained multifactor authenticator.

I use an authentication provider, what can I do?

If possible, setup your authentication provider to have UV required. This will cause some credentials to no longer work in your environment, but it will ensure that every authenticator has a consistent experience. In most cases, your authentication provider is likely to be standards compliant, and will not perform the extended verification discussed below meaning that “preferred” is bypassable, and “discouraged” can have inconsistent UV requests from users.

What can an RP do?

Because of this change, there are really only three workflows now that are actually consistent for users where we can enforce UV properties are observed correctly.

  • Required - At registration and authentication UV is always required
  • Preferred + with UV - At registration and authentication UV is always required
  • Preferred + no UV - At registration and authentication UV should not be required

The way that this is achieved is an extension to the Webauthn specification. When you register a credential you must store the state of the UV boolean at registration, and you must store the policy that was requested at registration. During authentication the following is checked in place of the webauthn defined UV check:

if credential.registration_policy == required OR authentication.policy == required {
    assert(authentication.uv == true)
} else if credential.registration_policy == preferred AND credential.registration_uv == true {
    // We either sent authentication.policy preferred or discouraged, but the user registered
    // with UV so we enforce that behaviour.
    assert(authentication.uv == true)
} else {
    // Do not check uv.
}

There is a single edge case in this work flow - since we now send “preferred” it’s possible that a credential that registered without UV (IE via Firefox which doesn’t support CTAP2.0_pre or greater) will be moved to using a platform that does support CTAP2.0_pre or greater, and it will begin to request UV. It is however possible in this scenario that once the credential begins to provide UV we can then store the credential.uv as true and enforce that for future authentications.

The primary issue with this is that we will begin to ask for the user’s PIN more often with credentials which may lead to frustration. Biometrics this is less of a concern as the “touch” action is always required anyway. However I think this is acceptable since it’s more important for a consistent set of behaviours to exist.

Previously I have stated that “preferred” should not be used since it is bypassable, but with the extensions to Webauthn above where policy and uv at registration are stored and validated, preferred gains a proper meaning and can be checked and enforced.

Conclusion

In the scenarioes where “discouraged” and “preferred” may be used, UV is meaningless in the current definition of the Webauthn specification when paired with the various versions of CTAP. It’s merely a confusing annoyance that we present to users seemingly at random, that is trivially bypassed, adds little to no security value and at worst undermines user trust in the systems we are trying to build.

When we are building authentication systems, we must always think about and consider the humans who will be using these systems, and the security properties that we actually provide in these systems.

Nextcloud - Unable to Open Photos Library

I noticed since macos 11.6.2 that Nextcloud has been unable to sync my photos library. Looking into this error in Console.app I saw:

error       kernel  System Policy: Nextcloud(798) deny(1) file-read-data /Users/william/Pictures/Photos Library.photoslibrary

It seems that Nextcloud is not sandboxed which means that macos enforces stricter permissions on what this can or can not access, which is what prevented the photos library from syncing.

To resolve this you can go to System Preferences -> Security and Privacy -> Privacy -> Full Disk Access and then grant Nextcloud.app full disk access which will allow it to read the filesystem.

I tried to allow this via the files and folders access but I was unable to add/remove new items to this list.

d i v c l a s s = " s e c t i o n " i d = " t r a n s a c t i o n a l - o p e r a t i o n s - i n - r u s t " >

Results from the OpenSUSE 2021 Rust Survey

From September the 8th to October the 7th, OpenSUSE has helped me host a survey on how developers are using Rust in their environments. As the maintainer of the Rust packages in SUSE and OpenSUSE it was important for me to get a better understanding of how people are using Rust so that we can make decisions that match how the community is working.

First, to every single one of the 1360 people who responded to this survey, thank you! This exceeded my expectations and it means a lot to have had so many people take the time to help with this.

All the data can be found here

What did you want to answer?

I had assistance from a psychology researcher at a local university to construct the survey and her help guided the structure and many of the questions. An important element of this was that the questions provided shouldn’t influence people into a certain answer, and that meant questions were built in a way to get a fair response that didn’t lead people into a certain outcome or response pattern. As a result, it’s likely that the reasons for the survey was not obvious to the participants.

What we wanted to determine from this survey:

  • How are developers installing rust toolchains so that we can attract them to OpenSUSE by reducing friction?
  • In what ways are people using distribution rust packages in their environments (contrast to rustup)?
  • Should our rust package include developer facing tools, or is it just another component of a build pipeline?
  • When people create or distribute rust software, how are they managing their dependencies, and do we need to provide tools to assist?
  • Based on the above, how can we make it easier for people to distribute rust software in packages as a distribution?
  • How do developers manage security issues in rust libraries, and how can this be integrated to reduce packaging friction?

Lets get to the data

As mentioned there were 1360 responses. Questions were broken into three broad categories.

  • Attitude
  • Developers
  • Distributors

Attitude

This section was intended to be a gentle introduction to the survey, rather than answering any specific question. This section had 413 non-answers, which I will exclude for now.

We asked three questions:

  • Rust is important to my work or projects (1 disagree - 5 agree)
  • Rust will become more important in my work or projects in the future.  (1 disagree - 5 agree)
  • Rust will become more important to other developers and projects in the future (1 disagree - 5 agree)
../../../_images/1.png ../../../_images/2.png ../../../_images/3.png

From this there is strong support that rust is important to individuals today. It’s likely this is biased as the survey was distributed mainly in rust communities, however, we still had 202 responses that were less than 3. Once we look at the future questions we see strong belief that rust will become more important. Again this is likely to be biased due to the communities the survey was distributed within, but we still see small numbers of people responding that rust will not be important to others or themself in the future.

As this section was not intended to answer any questions, I have chosen not to use the responses of this section in other areas of the analysis.

Developers

This section was designed to help answer the following questions:

  • How are people installing rust toolchains so that we can attract them to OpenSUSE by reducing friction?
  • In what ways are people using distribution rust packages in their environments (contrast to rustup)?
  • Should our rust package include developer facing tools, or is it just another component of a build pipeline?

We asked the following questions:

  • As a developer, I use Rust on the following platforms while programming.
  • On your primary development platform, how did you install your Rust toolchain?
  • The following features or tools are important in my development environment (do not use 1 - use a lot 5)
    • Integrated Development Environments with Language Features (syntax highlight, errors, completion, type checking
    • Debugging tools (lldb, gdb)
    • Online Documentation (doc.rust-lang.org, docs.rs)
    • Offline Documentation (local)
    • Build Caching (sccache)

Generally we wanted to know what platforms people were using so that we could establish what people on linux were using today vs what people on other platforms were using, and then knowing what other platforms are doing we can make decisions about how to proceed.

../../../_images/4.png

There were 751 people who responded that they were a developer in this section. We can see Linux is the most popular platform used while programming, but for “Linux only” (derived by selecting responses that only chose Linux and no other platforms) this number is about equal to Mac and Windows. Given the prevalence of containers and other online linux environments it would make sense that developers access multiple platforms from their preferred OS, which is why there are many responses that selected multiple platforms for their work.

../../../_images/5.png

From the next question we see overwhelming support of rustup as the preferred method to install rust on most developer machines. As we did not ask “why” we can only speculate on the reasons for this decision.

../../../_images/6.png

When we isolate this to “Linux only”, we see a slight proportion increase in package manager installed rust environments, but there remains a strong tendancy for rustup to be the preferred method of installation.

This may indicate that even within Linux distros with their package manager capabilities, and even with distributions try to provide rapid rust toolchain updates, that developers still prefer to use rust from rustup. Again, we can only speculate to why this is, but it already starts to highlight that distribution packaged rust is unlikely to be used as a developer facing tool.

../../../_images/7.png ../../../_images/8.png ../../../_images/9.png

Once we start to look at features of rust that developers rely on we see a very interesting distribution. I have not included all charts here. Some features are strongly used (IDE rls, online docs) where others seem to be more distributed in attitude (debuggers, offline docs, build caching). From the strongly supported features when we filter this by linux users using distribution packaged rust, we see a similar (but not as strong) trend for importance of IDE features. The other features like debuggers, offline docs and build caching all remain very distributed. This shows that tools like rls for IDE integration are very important, but with only a small number of developers using packaged rust as developers versus rustup it may not be an important area to support with limited packaging resources and time. It’s very likely that developers who are on other distributions, mac or windows are more comfortable with a rustup based installation process.

Distributors

This section was designed to help answer the following questions:

  • Should our rust package include developer facing tools, or is it just another component of a build pipeline?
  • When people create or distribute rust software, how are they managing their dependencies, and do we need to provide tools to assist?
  • Based on the above, how can we make it easier for people to distribute rust software in packages as a distribution?
  • How do developers manage security issues in rust libraries, and how can this be integrated to reduce packaging friction?

We asked the following questions:

  • Which platforms (operating systems) do you target for Rust software
  • How do you or your team/community build or provide Rust software for people to use?
  • In your release process, how do you manage your Rust dependencies?
  • In your ideal workflow, how would you prefer to manager your Rust dependencies?
  • How do you manage security updates in your Rust dependencies?
../../../_images/10.png

Our first question here really shows the popularity of Linux as a target platform for running rust with 570 out of 618 responses indicating they target Linux as a platform.

../../../_images/11.png

Once we look at the distribution methods, both building projects to packages and using distribution packaged rust in containers fall well behind the use of rustup in containers and locally installed rust tools. However if we observe container packaged rust and packaged rust binaries (which likely use the distro rust toolchains) we have 205 uses of the rust package out of 1280 uses, where we see 59 out of 680 from developers. This does indicate a tendancy that the rust package in a distribution is more likely to be used in a build pipeline over developer use - but rustup still remains most popular. I would speculate that this is because developers want to recreate the same process on their development systems as their target systems which would likely involve rustup as the method to ensure the identical toolchains are installed.

The next questions were focused on rust dependencies - as a staticly linked language, this changes the approach to how libraries can be managed. To answer how we as a distribution should support people in the way they want to manage libraries, we need to know how they use it today, and how they would ideally prefer to manage this in the future.

../../../_images/12.png ../../../_images/13.png

In both the current process and ideal processes we see a large tendancy to online library use from crates.io, and in both cases vendoring (pre-downloading) comes in second place. Between the current process and ideal process, we see a small reduction in online library use to the other options. As a distribution, since we can not provide online access to crates, we can safely assume most online crates users would move to vendoring if they had to work offline for packaging as it’s the most similar process available.

../../../_images/14.png ../../../_images/15.png

We can also look at some other relationships here. People who provide packages still tend to ideally prefer online crates usage, with distribution libraries coming in second place here. There is still significant momentum for packagers to want to use vendoring or online dependencies though. When we look at ideal management strategies for container builds, we see distribution packages being much less popular, and online libraries still remaining at the top.

../../../_images/16.png

Finally, when we look at how developers are managing their security updates, we see a really healthy statistic that many people are using tools like cargo audit and cargo outdated to proactively update their dependencies. Very few people rely on distribution packages for their updates however. But it remains that we see 126 responses from users who aren’t actively following security issues which again highlights a need for distributions who do provide rust packaged software to be proactive to detect issues that may exist.

Outcomes

By now we have looked at a lot of the survey and the results, so it’s time to answer our questions.

  • How are people installing rust toolchains so that we can attract them to OpenSUSE by reducing friction?

Developers are preferring the use of rustup over all other sources. Being what’s used on linux and other platforms, we should consider packaging and distributing rustup to give options to users (who may wish to avoid the curl | sh method.) I’ve already started the process to include this in OpenSUSE tumbleweed.

  • In what ways are people using distribution rust packages in their environments (contrast to rustup)?
  • Should our rust package include developer facing tools, or is it just another component of a build pipeline?

Generally developers tend strongly to rustup for their toolchains, where distribution rust seems to be used more in build pipelines. As a result of the emphasis on online docs and rustup, we can likely remove offline documentation and rls from the distribution packages as they are either not being used or have very few users and is not worth the distribution support cost and maintainer time. We would likely be better to encourage users to use rustup for developer facing needs instead.

To aid this argument, it appears that rls updates have been not functioning in OpenSUSE tumbleweed for a few weeks due to a packaging mistake, and no one has reported the issue - this means that the “scream test” failed. The lack of people noticing this again shows developer tools are not where our focus should be.

  • When people create or distribute rust software, how are they managing their dependencies, and do we need to provide tools to assist?
  • Based on the above, how can we make it easier for people to distribute rust software in packages as a distribution?

Distributors prefer cargo and it’s native tools, and this is likely an artifact of the tight knit tooling that exists in the rust community. Other options don’t seem to have made a lot of headway, and even within distribution packaging where you may expect stronger desire for packaged libraries, we see a high level of support for cargo directly to manage rust dependencies. From this I think it shows that efforts to package rust crates have not been effective to attract developers who are currently used to a very different workflow.

  • How do developers manage security issues in rust libraries, and how can this be integrated to reduce packaging friction?

Here we see that many people are proactive in updating their libraries, but there still exists many who don’t actively manage this. As a result, automating tools like cargo audit inside of build pipelines will likely help packagers, and also matches their existing and known tools. Given that many people will be performing frequent updates of their libraries or upstream releases, we’ll need to also ensure that the process to update and commit updates to packages is either fully automated or at least has a minimal hands on contact as possible. When combined with the majority of developers and distributors prefering online crates for dependencies, encouraging people to secure these existing workflows will likely be a smaller step for them. Since rust is staticly linked, we can also target our security efforts at leaf (consuming) packages rather than the libraries themself.

Closing

Again, thank you to everyone who answered the survey. It’s now time for me to go and start to do some work based on this data!