Of twenty sources in a reading list, ten came back as finished citations and ten were handed back. Handing them back is the right behaviour. This is what to do with them, which differs by cause — and only one of the three cases is solved by opening the page in a browser.
Anyone assembling a reference list from web sources meets the same wall eventually: the automated part stops, and it is not obvious whether the tool failed, the page is defended, or there was never anything there to collect. The three look identical from the outside — an empty result — and they need completely different work. Guessing wrong wastes an afternoon on a page that will never yield a record, or gives up on one that opens on the first click.
So the useful question is not how do I get a citation. It is which of the three is this?
1 of the twenty. A server-side reader gets 403; a browser gets the page. This is a bot defence, and it is the only case where opening the source yourself changes what is available.
What to do: open it in the browser you already have. Your session, your network, your institutional access. Then take the source with you before it changes — a full-page capture writes the page as one PDF with the URL and the retrieval date on it, which is what a reference to a web source has to carry anyway. The metadata for the entry is then in the header of your own file rather than in a service's index.
Why bother capturing at all, rather than noting the link: because we measured what happens to web sources after they are cited. A URL in a reference list is a promise about a page you no longer control.
4 of the twenty. 403 to a browser user agent as readily as to a reader. Here the client is not the problem: requests coming from a data centre are refused whatever they claim to be. Publishers do this to deter bulk downloading, and it catches every automated tool equally.
What to do: nothing clever. Open the page from your own connection, where these same publishers answer normally, and use their own export — most journal pages offer Cite → RIS or BibTeX, and that file is better than anything a reader can reconstruct. If you need the article itself rather than the entry, that is a library question, not a tooling question.
The one thing not to do is retry from the same place with a different user agent. It does not work, and a tool that pretends to be a browser to get past a rule that is aimed at it is a tool you cannot cite in good conscience.
5 of the twenty — the largest group, and the one people expect least. Fifty to ninety kilobytes of readable HTML, no defence of any kind, and no author, no date, no title of a work. A statistics portal page, a chamber-of-commerce service page, a news article, a software release.
What to do: write the entry yourself, because the decision the machine cannot make is what the work is. Is the source the statistics portal page, the dataset behind it, or the release the portal announces? A citation tool that answers here has picked one for you without saying so.
Two things make that manual entry defensible. First, the retrieval date, which for a page with no publication date is the only date the reference can carry. Second, the state of the page as you saw it — a capture stamped with the URL and the date, kept with the work. For grey literature and official web pages this is not belt and braces; a corporate page or an agency portal is rebuilt on no schedule and with no notice.
Two requests answer it. If the second succeeds where the first fails, it is
case 1. If both fail, case 2. If both succeed and the record still comes back
complete: false, case 3.
curl -sI -A 'my-reader/1.0' "$URL" | head -1 # as a reader
curl -sI -A "$BROWSER_UA" "$URL" | head -1 # as a browser
In an AI workflow the same three cases fall out of the record itself: a
warning naming a wall is case 1 or 2, and
complete: false with no warning is case 3. Which is why the flag
matters more than the title — the
measurement has two
records that carry a title, an author, and no completeness.