Linux package repositories are temporarily down (Debian and RPM packages) #4122

dagood · 2020-01-14T22:47:14Z

This looks like this on Ubuntu 19.04, during apt-get update:

E: Failed to fetch https://packages.microsoft.com/ubuntu/19.04/prod/dists/disco/main/binary-amd64/Packages.gz  File has unexpected size (29008 != 25211). Mirror sync in progress? [IP: 13.91.48.226 443]
   Hashes of expected file:
    - Filesize:25211 [weak]
    - SHA512:9c24d870e05029726895af3f19c91cb1f300e0406e0dafc4c77868f9d0b7da93b7546546f96d73411a59661fe787e8ae70da8db499b5d0d9f9b115ace9c64d49
    - SHA256:df8e493a9716e39b680ac89ce4ad189ceb00bd78ceded6bfb50cea2f0861d0e6
    - SHA1:7d4c121da39d6322b10caa73d37caeac667799a1 [weak]
    - MD5Sum:c428296bc9b29cc7237c751dbb0ffe9d [weak]
   Release file created at: Fri, 13 Dec 2019 17:48:48 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

This is expected to some degree whenever we push a Debian or RPM package to the feeds. I'm opening this issue to track it and raise awareness. It's possible there's an underlying issue here beyond the normal delay and I'm looking into it.

I have dotnet/docs#16670 open to add some description of the problem to the docs to make it easier to tell when this is happening and how long you normally need to wait.

/cc @dleeapho @richlander @NikolaMilosavljevic

Here's an all-up status report of what works for me:

Ubuntu 16.04 ✔️
Ubuntu 18.04 ✔️
Ubuntu 19.04 ✔️
Debian 9 ✔️ (up at 2020-01-15 00:42:35Z) https://icm.ad.msft.net/imp/v3/incidents/details/169363190/home
Debian 10 ✔️ (up at 2020-01-15 00:42:35Z) https://icm.ad.msft.net/imp/v3/incidents/details/169363190/home

On the RPM side, in particular CentOS 7, ✔️ (up at 2020-01-15 00:58:37Z or earlier) #4121.

Note that 2.1.15 is having trouble on the publish side and is tracked at #4124. The above is only for 3.0.2 and 3.1.1.

The text was updated successfully, but these errors were encountered:

dagood · 2020-01-14T23:09:34Z

Last I heard, the expected availability time is:

Debian package (apt-get): 30 minutes
RPM package (yum, zypper, dnf): 2 hours

Debian packages were published more than 3 hours ago, so I'm filing an internal ticket to get help with those.

RPM packages finished later, and there are ongoing deeper issues there. See #4121, "yum install dotnet-sdk-3.0 fails on centos7"

dagood · 2020-01-14T23:20:08Z

I filed https://icm.ad.msft.net/imp/v3/incidents/details/169350659/home for the Ubuntu 16.04/18.04/19.04 + Debian 9/10 issue. (Microsoft internal access required.)

rolinge · 2020-01-14T23:48:04Z

Docker builds are broken, since they use the debian repo

Logs

Step 4/20 : RUN wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg && mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/ && wget -q https://packages.microsoft.com/config/debian/9/prod.list && mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
---> Using cache
---> 9abb9d4016fa
Step 5/20 : RUN chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg && chown root:root /etc/apt/sources.list.d/microsoft-prod.list && apt-get install -y apt-transport-https dirmngr gnupg ca-certificates && apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
---> Using cache
---> a2750cab1fbf
Step 6/20 : RUN echo "deb https://download.mono-project.com/repo/debian stable-stretch main" | tee /etc/apt/sources.list.d/mono-official-stable.list
---> Using cache
---> 66b303a74706
Step 7/20 : RUN apt-get update
---> Running in 44e44ad16f9a
Hit:1 http://security.debian.org/debian-security stretch/updates InRelease
Ign:2 http://deb.debian.org/debian stretch InRelease
Hit:3 http://deb.debian.org/debian stretch-updates InRelease
Hit:4 http://deb.debian.org/debian stretch Release
Get:5 https://download.mono-project.com/repo/debian stable-stretch InRelease [5878 B]
Get:6 https://packages.microsoft.com/debian/9/prod stretch InRelease [4009 B]
Get:8 https://download.mono-project.com/repo/debian stable-stretch/main amd64 Packages [49.4 kB]
Get:9 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages [68.9 kB]
Err:9 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages
Writing more data than expected (72399 > 68862)
Hashes of expected file:

Filesize:68862 [weak]
SHA512:c45a7156e5d4796ec88721f1385bdff78e7f0396031c008a31b75cf5faecdb726a52a428256d55c9424af16d23c939f24fbf2a0d467c7082c7cdcbfad4b323c2
SHA256:5bccd1a4be6d9e2b60465cad6bcc793d0bd1f60a29de8fe2239453b528f8cf20
SHA1:600a5c433ae084819b5efa743ec9ec0ea38dc130 [weak]
MD5Sum:964348da7a5707ac25aaca7577e46728 [weak]
Release file created at: Mon, 16 Dec 2019 21:40:05 +0000
Fetched 59.3 kB in 1s (58.9 kB/s)
Reading package lists...
E: Failed to fetch https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.bz2 Writing more data than expected (72399 > 68862)
Hashes of expected file:
- Filesize:68862 [weak]
- SHA512:c45a7156e5d4796ec88721f1385bdff78e7f0396031c008a31b75cf5faecdb726a52a428256d55c9424af16d23c939f24fbf2a0d467c7082c7cdcbfad4b323c2
- SHA256:5bccd1a4be6d9e2b60465cad6bcc793d0bd1f60a29de8fe2239453b528f8cf20
- SHA1:600a5c433ae084819b5efa743ec9ec0ea38dc130 [weak]
- MD5Sum:964348da7a5707ac25aaca7577e46728 [weak]
Release file created at: Mon, 16 Dec 2019 21:40:05 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
The command '/bin/sh -c apt-get update' returned a non-zero code: 100

dagood · 2020-01-14T23:49:49Z

@rolinge I put the logs in a collapsible section to save some scrolling, hope that's ok.

The Ubuntu issue has been resolved by the linux repo team, and yeah, I was trying this out on various distros and saw the Debian 9 and Debian 10 error.

I've filed a new ticket for Debian 9/10: https://icm.ad.msft.net/imp/v3/incidents/details/169363190/home

wenxuan-f · 2020-01-14T23:50:20Z

It is working now! woohoo TY

rolinge · 2020-01-14T23:55:09Z

@rolinge I put the logs in a collapsible section to save some scrolling, hope that's ok.

The Ubuntu issue has been resolved by the linux repo team, and yeah, I was trying this out on various distros and saw the Debian 9 and Debian 10 error.

I've filed a new ticket for Debian 9/10: https://icm.ad.msft.net/imp/v3/incidents/details/169363190/home

@dagood when you have a minute, show me how to do do the 'collapsible section' trick 😄
Never mind, I found it...

A collapsible section with markdown

Like This!

stuff
more stuff

rolinge · 2020-01-15T00:33:26Z

My docker build is proceeding, so it appears to be fixed for debian 9.
Thanks!

dagood · 2020-01-15T00:43:46Z

Yep, they fixed up Debian 9 and 10, updated my list. Thanks for checking. 😄

harshavardhana · 2020-01-15T00:48:23Z

How can we ensure that this doesn't happen again? - This is this the third time in the last 1yr and whenever it happens the entire project CI/CD breaks because we are dependent on it unfortunately.

I am sure many are with me here - I hope there are more stricter guidelines on verifying what is being published. Publishing packages should be simple and shouldn't break things in this weird manner.

dagood · 2020-01-15T01:23:40Z

How can we ensure that this doesn't happen again?

We (.NET Core) are trying to work internally with the team that owns these package repos to improve this. packages.microsoft.com is a shared service that a bunch of Microsoft products (VS Code, PowerShell) rely on to be available on Linux through package managers.

I am sure many are with me here - I hope there are more stricter guidelines on verifying what is being published. Publishing packages should be simple and shouldn't break things in this weird manner.

Unfortunately the problem isn't what's being published--I wish it were, so we could take ownership of some fix--rather, it's with the infrastructure behind the feeds. It is super frustrating that it's "normal" to have an up to 2hr service outage every time we publish a package (or someone else publishes a package), and that we need to effectively monitor it ourselves and ping the team when it doesn't come back online.

dagood · 2020-01-15T01:24:44Z

It looks like all the known issues are resolved now except for 2.1.15 specifically, which is being tracked at #4124. Closing as resolved, thanks for your patience.

esetnik · 2020-01-21T18:22:27Z

It appears to be broken again

Reading package lists...
E: Failed to fetch https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.bz2  File has unexpected size (72896 != 72828). Mirror sync in progress? [IP: 40.76.35.62 443]
   Hashes of expected file:
    - Filesize:72828 [weak]
    - SHA512:3e2135a607d82f3d0fb22a952998f1306ce456553daedb681bee125507c96d0b6e6429b110c92ae9cef8ae09974c74fbabb4e28aeab32ebed9962d2c61238686
    - SHA256:4217b6ad3a63657ec03df9a3f95c0c5a55401f760790b48269e80dcf6ac19171
    - SHA1:5b709f4947671085a4172e9d90248fef90be7355 [weak]
    - MD5Sum:3564888127c08fa04b2b27e345507089 [weak]
   Release file created at: Tue, 21 Jan 2020 18:06:13 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

Marche86 · 2020-01-21T18:37:29Z

I have the same problem right now.

esetnik · 2020-01-21T18:38:34Z

For reference see my support forum ticket https://social.msdn.microsoft.com/Forums/sqlserver/en-US/455e0216-aa0f-4d57-9409-776aea3407f0/failed-to-fetch?forum=sqldriverforphp#455e0216-aa0f-4d57-9409-776aea3407f0

esetnik · 2020-01-21T18:43:24Z

@dagood do you want to reopen this or should I create a new issue?

dagood · 2020-01-21T18:45:39Z

Thanks for reporting, I've opened #4165. I'm not currently able to repro it so it's probably that expected delay after someone pushed a package. Still checking to make sure I'm not doing something wrong.

dagood · 2020-01-21T18:53:57Z

It appears to have been an expected outage because some packages were pushed by a team that shares this repository. Let me know on #4165 if it still repros for you.

eusebiu · 2020-04-29T06:02:16Z

I get this for 19.10 installing dotnet core SDK 3.1 on Linux Mint 19.3 on sudo apt-get update:
Err:14 https://packages.microsoft.com/ubuntu/19.10/prod bionic Release
404 Not Found [IP: 13.81.215.193 443]
Err:15 https://packages.microsoft.com/ubuntu/19.04/prod bionic Release
404 Not Found [IP: 13.81.215.193 443]

dagood added the area-setup Issues related to installing .NET Core label Jan 14, 2020

dagood mentioned this issue Jan 14, 2020

.NET Core Linux Feeds -- Now available #3995

Closed

lucaspimentel mentioned this issue Jan 14, 2020

Can't install .NET Core on Ubuntu, Debian from the Linux package repository: "File has unexpected size (23655 != 156065)" #3988

Closed

This was referenced Jan 14, 2020

Change Linux distro for CI dotnet/vscode-csharp#3509

Closed

Add help link to .NET SDK install prompt dotnet/vscode-csharp#3508

Merged

dagood closed this as completed Jan 15, 2020

rbhanda mentioned this issue Jan 16, 2020

.NET Core January 2020 Update - 2.1.15, 3.0.2 and 3.1.1 #4119

Closed

5 tasks

dagood mentioned this issue Jan 21, 2020

Debian 9 Linux package repository unavailable, "Mirror sync in progress?" #4165

Closed

vkuehn mentioned this issue Jun 17, 2021

Can not install msodbcsql17 package distribution #6386

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linux package repositories are temporarily down (Debian and RPM packages) #4122

Linux package repositories are temporarily down (Debian and RPM packages) #4122

dagood commented Jan 14, 2020 •

edited

Loading

dagood commented Jan 14, 2020

dagood commented Jan 14, 2020 •

edited

Loading

rolinge commented Jan 14, 2020 •

edited by dagood

Loading

dagood commented Jan 14, 2020

wenxuan-f commented Jan 14, 2020

rolinge commented Jan 14, 2020 •

edited

Loading

rolinge commented Jan 15, 2020

dagood commented Jan 15, 2020

harshavardhana commented Jan 15, 2020

dagood commented Jan 15, 2020

dagood commented Jan 15, 2020

esetnik commented Jan 21, 2020

Marche86 commented Jan 21, 2020

esetnik commented Jan 21, 2020

esetnik commented Jan 21, 2020

dagood commented Jan 21, 2020

dagood commented Jan 21, 2020

eusebiu commented Apr 29, 2020 •

edited

Loading

Linux package repositories are temporarily down (Debian and RPM packages) #4122

Linux package repositories are temporarily down (Debian and RPM packages) #4122

Comments

dagood commented Jan 14, 2020 • edited Loading

dagood commented Jan 14, 2020

dagood commented Jan 14, 2020 • edited Loading

rolinge commented Jan 14, 2020 • edited by dagood Loading

dagood commented Jan 14, 2020

wenxuan-f commented Jan 14, 2020

rolinge commented Jan 14, 2020 • edited Loading

A collapsible section with markdown

rolinge commented Jan 15, 2020

dagood commented Jan 15, 2020

harshavardhana commented Jan 15, 2020

dagood commented Jan 15, 2020

dagood commented Jan 15, 2020

esetnik commented Jan 21, 2020

Marche86 commented Jan 21, 2020

esetnik commented Jan 21, 2020

esetnik commented Jan 21, 2020

dagood commented Jan 21, 2020

dagood commented Jan 21, 2020

eusebiu commented Apr 29, 2020 • edited Loading

dagood commented Jan 14, 2020 •

edited

Loading

dagood commented Jan 14, 2020 •

edited

Loading

rolinge commented Jan 14, 2020 •

edited by dagood

Loading

rolinge commented Jan 14, 2020 •

edited

Loading

eusebiu commented Apr 29, 2020 •

edited

Loading