logo

 
Cycling Routes
 
 
blog.rob.sh
A Sad Day
At lunchtime today I was told via IRC/e-mail that Chris Orme, of Datahop passed away after a flight back from a peering conference in Florida. I've worked with Chris in the past, and despite having some differences of opinion, found him to be a great guy to work with. We had a disagreement about some contractual matters, and Chris was man enough to apologise to me in person at the LONAP 10th Birthday event. I was happy that we'd resolved our differences, and we exchanged contact details again. I was looking forward to working with him in the future.

This blog post doesn't really have much relevance to anyone else, other than to say that my thoughts are with Chris' family, and his girlfriend, Nat. I'm hoping that someone can suggest a cause that Chris cared about, so that donations can be made in his memory.

Rest in peace, Chris.
Tagged in: ISP, LINX
IPv6 - It Doesn't Just Work.
I was reading an entry posted by Brett Carr on Nominet's techblog today entitled "ipv6 It just works". Unfortunately, for IPv6, and for the sentiment behind this message (IPv6 can be run pretty easily!), in my experience, IPv6 - it doesn't just work!

It's easy to dismiss the previous sentence, given that many networks aren't designed to run IPv6, and there's kit out there that's just not IPv6-capable yet. When building the AS29636 network, we specified that IPv6-capability was one of the things that would be a requirement of the kit that was going into the new network, not just something that we'd like to have. We work to a similar specification at my current employer, - which ensures that we can deploy IPv6 within a pre-agreed timeframe once we have some commercial drive for it (either from customers, or for business continuity reasons). I think that this is the best way for a SP network to be a the moment - there's no revenue in having IPv6 deployed (generally), but there might be lost revenue when a customer comes to your network with IPv6 as a requirement in their RFQ...

Returning to the reason that I started writing this post - the problems for IPv6 deployment don't just come from the fact that your hardware doesn't necessarily support it, and it isn't just that running IPv6 on your kit might have financial implications for the software licensing that you're going to be deploying (the arbitrary Cisco requirement for advipservices for IPv6 is a completely separate post). There are going to be issues where you don't necessarily expect them - which can be hard to debug, where IPv6 "should just work", it doesn't.

Without mentioning any specifics of a case that was brought to my attention in the last couple of weeks - a customer was having problems getting IPv6 traffic flowing across a layer 2 ethernet circuit. The expectation of this circuit that you can put ethernet frames onto it (and it doesn't really matter what the ethertype is, just that they're valid ethernet frames) - and they are going to be punted down the link, to whatever you terminate the L2 circuit on. With IPv4, this not working would be a disastrous failure - the product just wouldn't be working. However, this particular circuit was not passing frames that contained IPv6 packets. As it turns out, the carrier's equipment in the path contained a firmware bug that was causing the frames containing IPv6-packets to be dropped - and hence, no neighbour-discovery, and no traffic flow between the two ends of the circuit.

This is just one isolated case - but the question is, where else in your network do you have a problem like this one? How much kit that may, right now, be considered something that shouldn't be interfering anywhere above L2, is going to exhibit this type of problem? How much load is this going to cause your NOC? How much time liasing with circuit suppliers, and telcos is going to be spent actually deploying IPv6 on your network? I think these questions are starting to form a basis of why SPs should be startng to roll out IPv6 onto your network now. The lack of transition plan from IPv4 to IPv6, and the fact that IPv6 hasn't had widespread deployment testing across many platforms and transmission media mean that deploying IPv6 in a rush across your network isn't necessarily going to be as easy as you've thought.

Whilst I applaud the fact that Nominet are ensuring that they're going to be ready to run the UK ccTLD with IPv6 nameservers, and that their infrastructure is ready - I don't think that IPv6 is going to be quite as easy to deploy as Brett found in his blog post.
Tagged in: Tech, ISP, IPv6
Building the RIPEDB server
It took me a few hours over the course of this week to build the RIPE whois server for some internal projects -- given that there seems to be a very limited amount of documentation for the build process, and threads on mailing lists, I'm going to post this here. I hope that it gets picked up by Google.

The first problem that is encountered is that the libtool that is included with the whois server does not support 'modern' tags, such as --tag=CC. This looks to be because the included libtool is somewhat dated. This can be easily fixed by using the system libtool:

[rjs@dbhost whoisserver-nightly]$ mv libtool libtool.old
[rjs@dbhost whoisserver-nightly]$ ln -s `which libtool` .
(NB: after doing this, you should specify --no-all --setup-db --setup-config --install --setup-tests, and _NOT_ --configure, otherwise the existing libtool will just replace the symlink that you've created)

The next problem is that there are a large number of definitions of yywrap() that conflict when they are being linked. On examination, these seem to be of the form:
int yywrap(){
    return 1;
}
There are definitions in:
  • src/modules/rpsl/syntax.c
  • src/modules/rpsl/mnt_routes.lex.c
  • src/modules/rpsl/mnt_routes6.lex.c
  • src/modules/rpsl/mnt_routes_an.lex.c
that conflict with each other. Simply removing the yywrap function that only returns 1 from each of these files resolves this linker issue.

The next problem is that there is a multiple definition of a 'set_dynamic' function -- I believe that this is a function that's used both in the src/modules/pc/pc_commands.c files, and in the MySQL headers.
/usr/lib/mysql/libmysqlclient_r.a(array.o): In function `set_dynamic':
/home/mysqldev/rpm/BUILD/mysql-4.1.22/libmysql_r/array.c:175: multiple definition of `set_dynamic'
/home/rjs/tmp/whoisserver-nightly/src/.././src/librip.a(pc_commands.o):
/home/rjs/tmp/whoisserver-nightly/src/modules/pc/pc_commands.c:509: first defined here
/usr/bin/ld: Warning: size of symbol `set_dynamic' changed from 366 in
/home/rjs/tmp/whoisserver-nightly/src/.././src/librip.a(pc_commands.o) to 203 in
/usr/lib/mysql/libmysqlclient_r.a(array.o)
The first definition (/home/mysqldev/rpm/BUILD/...) is from the RPM package of the shared MySQL libraries. I've tried the compatibility libraries, as well as versions from MySQL 4.1, and MySQL 4.0. The only way I can find to correct this is to change where the include path is from the files that are provided by the MySQL development libraries, to the shared libraries provided by MySQL:
[rjs@dbhost whoisserver-nightly]$ diff Makefile Makefile.orig 
214c214
< MYSQL_LIBS = -L/usr/lib/ -lmysqlclient_r -lz -lcrypt -lnsl -lm
---
> MYSQL_LIBS = -L/usr/lib/mysql -lmysqlclient_r -lz -lcrypt -lnsl -lm
[rjs@dbhost whoisserver-nightly]$ diff src/Makefile src/Makefile.orig 
378c378
< MYSQL_LIBS = -L/usr/lib/ -lmysqlclient_r -lz -lcrypt -lnsl -lm
---
> MYSQL_LIBS = -L/usr/lib/mysql -lmysqlclient_r -lz -lcrypt -lnsl -lm
This then allows the whoisd to compile.

However, when starting the server a number of segmentation faults are experienced:
[rjs@dbhost bin]$ ./whoisd_start --config=rip.config --crashes=1
Starting whois-server daemon with configuration
/home/rjs/whoistmp//conf/rip.config from /home/rjs/whoistmp//bin
./whoisd_start: line 165: 23118 Segmentation fault      (core dumped) $NOHUP_NICENESS $WHOISRIP -p $pid_file -c ${CONFIG} >> $err_log 2>&1
mv: cannot stat `core': No such file or directory
./whoisd_start: line 165: 23145 Segmentation fault      (core dumped) $NOHUP_NICENESS $WHOISRIP -p $pid_file -c ${CONFIG} >> $err_log 2>&1
mv: cannot stat `core': No such file or directory
081008 11:24:36  $WHOISD ended
This is because the MySQL connections do not work correctly out of the box -- what you will need to do is to go to src/SQL/ and create the DB manually:
[rjs@dbhost SQL]$ mysql -utest_db -pPASSWORD test_db < create.tables.sql
[rjs@dbhost SQL]$ mysql -utest_db -pPASSWORD test_db < main.index.1
As long as your $PREFIX/conf/sources.config is correct (correct U/P for the object database, not the admin db), and the rip.conf has the admin DB specified correctly -- then the server should then start.

This was originally going to be an e-mail to ripe-dbm, but I seem to have fixed it during the course of writing the mail!

Tagged in: Tech, Work, ISP, RIPE
Last Friday, Andy Davidson, Jonathan Oddy, and I pushed out some research that has some quite worrying repercussions. Whilst I've heard from a lot of people privately about this matter, there's a big flaw here, and as Andy posted on his blog (which is much more informative than mine, I think!), this is a big problem.

The reason, I think, that we're getting limited public discussion of this exploit (I hesitate to call it an exploit, it's a flaw really, because it's actually a result of the RFC that the problem exists), is because the implementations of 4-byte AS support that are out there already are generally not standards compliant. Let's run down the list:
  • Juniper - not standards compliant on this matter, some form of AS4_PATH and AS_PATH munge happens during processing, and the paths don't come out with the confeds in. I've verified this by looking at the updates via a Goscomb Technologies Juniper box - look at the monitor traffic output.
  • Force10 - Greg Hankins has a test peering box running at route-server.cluepon.net, which shows the following
    Received from : 
      72.37.255.12  (72.37.255.1)    Best
        AS_PATH :  18508 19151 35320 196629 23456
    
    This looks to be doing the same thing as JunOS is doing, and ignoring the invalid parts of the AS4_PATH when building the AS_PATH.
  • Cisco - IOS XR/XE - I've not yet confirmed this one, but I believe this doesn't have a standards compliant implementation either. I'm hoping to get some time with someone to discuss what they're seeing later tonight, or maybe later this week. I think that, given that we haven't heard anything, and there are likely to be _some_ SPs that are learning 91.207.218.0/23 via 35320, this is probably another platform that doesn't comply.
  • Cisco - IOS - There's just one lonely IOS release that runs AS4 right now, and that's 12.0(32)S12. This is the IOS that we used to demonstrate how IOS reacts to the flaw in the RFC. When IOS sees AS_CONFED_SET in the AS4_PATH, it drops the session, as required by the RFC

Maybe there are a couple of interesting points here, why are most vendors not actually complying with this RFC, does this mean that they've spotted what Andy, Jonathan and I have reported on, and dropped this requirement? If this is the case, then I wonder, when IETF IDR is so full of people with @juniper.net, and @cisco.com addresses - why did this ever appear in the first place?

This is a serious flaw in the standards, and despite the fact that today, we reported on how the issue has actually come to pass, this is going to remain open, unless we fix the RFC.

The issue here is, with most (if not all) BGP attributes, there's almost an expectation that the immediate neighbour will sanity-check what their peer has sent - if it's one hop away, you can generally interact directly with that neighbour, and work out what the problem is, there's no-one harmed, as just one session is dropped, by two networks sharing some adjacency. A case in point of this, is the problem that we saw with Cisco not obeying the RFC relating to sending UPDATES before KeepAlives in BGP conversations (CSCsu84268). As far as I saw, this bug only affected directly connected neighbours, and hence there was no major impact. Now, let's consider what happens the case of the AS4_PATH problem we reported. AS4_PATH is optional transitive in BGP, hence, if you hand it to a non-AS4 speaker, the router will just transmit it along to the peers it advertises the route to. This is a reasonably neat solution, one would think, as AS4 information is transmitted, but it doesn't require every router in the path between two AS4 speakers to understand it, yet they can still get the same information as they could if the path was completely made up of AS4 speakers. Furthermore, by appending 23456 to AS_PATH, then even the non-AS4 speakers understand that there was some AS in this path. However, this also means that if I announce, to a non-AS4 speaker, a completely invalid AS4_PATH, they don't know anything about it, or the contents, and hence can't sanity check it. This results in me being able to tunnel my AS4_PATH across the internet.

Great, so now I've described, in some more chatty language, what we wrote on NANOG, and C-NSP. What does this mean to any operator? Well, if I take a prefix, originate it, and then announce it to the internet, then I can get the first AS4 speaker I find to tear down whatever session they learned my prefix on. If I combine this with injecting some ASNs into the path, so that some networks don't accept it (due to loop prevention), then I can probably work out a way to get _my_ copy of the update across to you. In IOS's logs, you can't even tell who originated that prefix, and it doesn't seem to show the whole AS_PATH/AS4_PATH either. Say I send two prefixes that you learn one via one transit provider, and the other via another, I'll disconnect your full table connectivity.

This isn't even a bug, this is a flaw in the standard. I'd really like to get this fixed, and the way to do that is to get a bunch of operator experience/views, and take it to the IETF. So, if this concerns you (or you're going to need to deploy a new point release -- like 12.0(32)S12 is, or maybe need hardware support with 12.2SRE...), please put some pressure on your vendor, or drop me a note at rjs@eng.gxn.net.
LINX 65 Presentation
Further to my previous post - I presented this issue at LINX65 - video and slides can be found below.

Video
Fixed Slides - LINX's PowerPoint install seems to have corrupted my slides on the day.




Comments and feedback are most welcome.
A quick personal post to break the silence here!

I'm currently very interested in hearing about any UK or EU-based network engineering or architecture opportunities that are out there, especially in SP networks that run MPLS with TE. If anyone has some such opportunity, or knows of something that they think might suit me -- please drop me a mail to rjs@rob.sh for a copy of my CV.

An outline of my CV is available on LinkedIn.

I'm hoping to find some time to put some technical articles together that can be posted here in the near future.
Tagged in: Tech, Work, ISP
I've had a couple of mails relating to this PSN, which again references the research that Andy Davidson, Jonathan Oddy and I did last year. It seems that some of the sources of the initial mailing list posts we made are gone (particularly the merit.edu one that is referenced from both Juniper's site and most other places). For that reason, I've included both the mails that we sent to NANOG/C-NSP/J-NSP last year here.


Date: Fri, 16 Jan 2009 12:57:19 +0000
From: Rob Shakir 
To: cisco-nsp@puck.nether.net, nanog@nanog.org
Subject: BGP Session Teardown due to AS_CONFED_SEQUENCE in AS4_PATH
Message-ID: <20090116125718.GB26415@bronze.eng.gxn.net>


Strict RFC 4893 (4-byte ASN support) BGP4 implementations are vulnerable to a
session reset by distant (not directly connected) ASes. This vulnerability is a
feature of the standard, and unless immediate action is taken an increasingly
significant number of networks will be open to attack. Accidental triggering of
this vulnerability has already been seen in the wild, although the limited
number of RFC 4893 deployments has limited its effect.  

Summary:
It is possible to cause BGP sessions to remotely reset by injecting invalid data
into the AS4_PATH attribute provided to store 4-byte ASN paths. Since AS4_PATH
is an optional transitive attribute, the invalid data will be transited through
many intermediate ASes which will not examine the content. To be vulnerable, an
operator does not have to be actively using 4-byte AS support. This problem was
first reported by Andy Davidson on NANOG in December 2008 [0], furthermore we
have been able to demonstrate that a device running Cisco IOS release
12.0(32)S12 behaves as per this description.

Details:

When a prefix is learnt from a BGP neighbour that does not support 4-byte ASNs,
the AS4_PATH attribute is retained, and appended to UPDATE messages sent to
other neighbours [1, 3]. RFC4893 specifies that AS_CONFED_SEQUENCE and
AS_CONFED_SET are invalid in an AS4_PATH, the intention of which is to ensure
that an AS with a mix of AS4-aware BGP speakers, and AS4-unaware BGP speakers
does not propagate confederation AS paths outside of the confederation [1, 3].
Upon receiving an invalid BGP UPDATE message, a BGP speaker must send a
NOTIFICATION message [2, 6.3], after a NOTIFICATION message, the BGP connection
is closed [2, 4.5].

Analysis of the Reported Path:   

On 10th December 2008, a BGP update was propagated with illegal/invalid
confederation attributes in the AS4_PATH.  When this update was received by AS4
aware BGP speakers, the RFCs described above were interpreted literally and the
session was torn down. Because the illegal attributes were learned on a transit
session, an affected network can have global reachability impaired.

Please note that the analysis of this path describes what we expect to have
happened in this case, it has not been confirmed by any of the ASNs involved.

91.207.218.0/23 
	Path Attributes - Origin: Incomplete 
	Flags: 0x40 (Well-known, Transitive, Complete) 
	Origin: Incomplete (2) 
	AS_PATH: xx xx 35320 23456 (13 bytes) 
	AS4_PATH: (65044 65057) 196629 (7 bytes) 

In this data, the AS_PATH indicates that a prefix is announced by an AS4 speaker
(as indicated by AS23456) and propagated through by AS35320. The AS4_PATH data
shows that the AS4 originator is AS196629, the rest of this path is an
AS_CONFED_SEQUENCE [3, 5]. It would appear that in this case, AS196629 peers
with AS35320, which is AS4-aware on this border. The prefix is then propagated
through AS35320, with the AS4 aware routers appending their ASN to the
AS_CONFED_SEQUENCE. This is in contravention of RFC 4893 [1, 3]. The border
which announces this route to AS35320's upstream does not appear to be
AS4-aware. During normal announcements, the BGP speaker on a border with an
upstream ASN that is not part of the confederation will remove the left-most
AS_CONFED_SETs or AS_CONFED_SEQUENCEs that exist in the AS_PATH [3, 6.1] and
replace them with the confederation identifier. However, due to the fact that
both AS_CONFED_SET and AS_CONFED_SEQUENCE are invalid in an AS4_PATH, then no
such action is taken on the border between an AS4 aware AS, and a non-AS4 aware
AS. In addition, since the AS35320 border is not AS4 aware, then it does not
update the AS4_PATH.

This malformed UPDATE is then sent to AS35320's upstream, if there are no
AS4-aware routers in the path between the AS35320 border, and an AS receiving
this update, the AS4_PATH will not have been analysed. The first AS4-aware
router to receive this update will reset the session towards the neighbour from
whom it receives the update. 

The border which announces this route to AS35320's upstream does not appear to
be AS4-aware; If it were a strict AS4 implementation it would reset the BGP
session due to the malformed AS4_PATH, and a broken implementation that treats
AS4_PATH as an equivalent of the AS_PATH would sanitise the AS4_PATH. This
allows the AS4_PATH containing an AS_CONFED_SET to be passed to neighbouring
networks.

This escape of an AS_CONFED_SET from a network with only partial AS4 support is
exactly the situation that RFC 4893 attempts to avoid by forbidding the presence
of an AS_CONFED_SET in the AS4_PATH. In the ideal world the neighbouring network
receiving an UPDATE containing this obviously malformed AS4_PATH would reset the
session, preventing further propagation and isolating the broken network.

Unfortunately the vast majority of networks do not support AS4 so pass on this
malformed AS4_PATH to their neighbours. The first AS4-aware router to receive
this update will reset the session towards the neighbour from whom it received
the update.

Cisco IOS Behaviour:

In a lab environment, a Cisco 7200 running IOS 12.0(32)S12, which is able to
support 4-byte ASNs, was peered with a Cisco 2811 running 12.4(19). When the BGP
session to the upstream 2811 is established by the 7200, the following log
messages are observed:

*Jan 16 11:29:58.531: %BGP-5-ADJCHANGE: neighbor 193.239.32.2 Up 
*Jan 16 11:30:02.595: %BGP-6-ASPATH: Invalid AS path (65044 65048 65062) 3.21 23456 received from 193.239.32.2: Confederation found in AS4_PATH
*Jan 16 11:30:02.595: %BGP-5-ADJCHANGE: neighbor 193.239.32.2 Down BGP Notification sent
*Jan 16 11:30:02.595: %BGP-3-NOTIFICATION: sent to neighbor 193.239.32.2 3/1 (update malformed) 27
 bytes E0111803 030000FE 140000FE 180000FE 26 FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 0050 0200 0000
 3540 0101 0240 020C 0205 3D25 2114 89F8 5BA0 5BA0 4003 04C1 EF20 02E0 1118 0303 0000 FE14 0000 
FE18 0000 FE26 0202 0003 0015 0000 5BA0 175B CFDA

The configuration on the 7200 is as follows:

router bgp 65123
no synchronization
bgp log-neighbor-changes
neighbor 193.239.32.2 remote-as 15653
no auto-summary

The BGP session will continue to be reset each time the invalid AS4_PATH is
received.

Possible Impact:

During a BGP conversation, it is expected that a neighbour's UPDATE messages are
sanitised by the immediate neighbour, during a 'normal' BGP conversation, if a
BGP speaker receives an invalid UPDATE, it will teardown the session, and this
invalid UPDATE will not propagate any further. In the case of optional
transitive attributes such as AS4_PATH, this invalid update can be transited
through many ASes, as the content of the invalid attribute in the UPDATE message
is not examined.

In a hypothetical scenario, an AS4 aware service provider (A) has a transit
provider (T) that is not AS4 aware. BGP speaker B, a large distance from A has a
bug affecting their equipment that introduces an AS_CONFED_SET in the AS4_PATH.
Since B's updates are propagated through to A via T, A will tear down the
session to T due to the malformed attribute. This is an out of proportion
reaction as the update may affect only one prefix in a full BGP table. If this
update is also propagated through A's other transit providers A may lose
full-table visibility until one of their transit providers filters the route.
Examining the UPDATE message to establish which route caused session teardown
may be a non-trivial activity.


Conclusion:

Whilst this description may be applied to invalid data in any optional
transitive element, it has a greater impact with AS4_PATH due to the large
number of BGP speakers that currently do not examine any 4-byte ASN data in an
UPDATE. There has been a discussion of this matter on the IETF IDR mailing list
[4], however, due to availability of Cisco IOS containing AS4 support
(12.0(32)S12), and an observation of this problem 'in the wild', we believe that
it is of operational concern to those that are planning on deployment of
AS4-aware platforms [5].

Any input from the operational community relating to this problem is much
appreciated, either publicly, or privately.

Regards,
	Andy Davidson, NetSumo (andy.davidson@netsumo.com),
	Jonathan Oddy, Hostway UK (jonathan.oddy@hostway.co.uk),
	Rob Shakir, GX Networks (rjs@eng.gxn.net)

References:
[0]: Andy Davidson - 91.207.218.0/23 prefix in DFZ - AS3.21 / AS196629 -
    announced with AS_CONFED_SEQUENCE in AS4_PATH - propagated by 35320,
    http://markmail.org/message/3ofvjyggayfxezna
[1]: rfc4893: BGP Support for Four-octet AS Number Space
[2]: rfc4271: A Border Gateway Protocol 4 (BGP-4)
[3]: rfc3054: Autonomous System Confederations for BGP
[4]: Kaliraj Vairavakkalai, Juniper Networks, [Idr] RFC-4893 handling malformed
    AS4_PATH attributes,
    http://www.ietf.org/mail-archive/web/idr/current/msg03368.html
[5]: http://as4.cluepon.net/index.php/Software_Support

Thanks to Will Hargrave (LONAP) for assistance with this document.


Date: Wed, 21 Jan 2009 10:14:24 +0000
From: Rob Shakir 
To: nanog@nanog.org
Subject: Re: BGP Session Teardown due to AS_CONFED_SEQUENCE in AS4_PATH
Message-ID: <20090121101424.GB5577@bronze.eng.gxn.net>
References: <20090116125718.GB26415@bronze.eng.gxn.net>

Hi,

Further to the initial research sent to NANOG, after discussions with a number
of operators, we have compiled some recommendations on the handling of invalid
AS4_PATH attributes. 

Any feedback on these recommendations is appreciated:

As discussed on the IETF IDR list last month, there are concerns relating to the
treatment of AS_CONFED_SET/SEQUENCE in AS4_PATH as described in RFC4893 [0].
Since the last post to that thread the situation has been made more urgent with
the release of Cisco IOS 12.0(32)S12, which responds to malformed AS4_PATH
attributes by sending a NOTIFICATION to the neighbour, and tearing down the BGP
adjacency. This behaviour seems to be required by RFC4721 section 6.3, as there
is no alternative error handling defined in RFC4893. As posted last Friday [1],
and discussed on the IDR list, this strict implementation introduces a new
attack vector by which a BGP session can be torn down due to a an attribute
populated by a distant BGP neighbour. These malformed attributes have already
been seen in the wild as a result of a error in Juniper's implementation of
RFC4893. 

Following discussions with a number of operators, we have attempted to generate
some recommendations relating to the behaviour that would be operationally most
useful when treating the invalid data in the AS4_PATH optional transitive
attribute.

There are two cases to consider when an invalid AS4_PATH is received:
  (1) A path to the prefix is not already known from that neighbour. 
  (2) A path to the prefix has already been learnt from that neighbour; 

In case (1) we recommend that the BGP speaker should discard the UPDATE and log
the fact. The log entry should include the received AS_PATH and
AS4_PATH to aid in debugging.

In case (2) we recommend that the BGP speaker should treat the UPDATE as a
withdrawal of existing path to the prefix. As per case (1) a log entry should be
raised to indicate that this has occurred.

It is quite possible that in both cases this behaviour may result in the BGP
speaker no longer having a valid path to the destination. We foresee that this
lack of a prefix in a BGP speaker's routing table may cause some operational
load initially, however, we feel that this is acceptable, considering the
alternate behaviours.

Should a prefix be injected into the global table with an invalid AS4_PATH, and
should the newly advertised (invalid) path be selected by all upstreams
available to a given ASN then this ASN will lose reachability to the prefix.
Whilst this can be abused we do not see this as more serious than the existing
possibility of malicious injection and blackholing of a prefix by a 3rd party.
As long as the rejection of paths due to invalid AS4_PATHs is clearly reported
to the administrator the source of the problem can be clearly identified. 

We consider that attempting to extract a valid AS4 or AS_PATH from the invalid
UPDATE is a mistake since this allows the propagation of invalid BGP data. In
addition, incorrect implementation of this comparatively complex mechanism by a
vendor may result in loops. By explicitly not installing prefixes with invalid
AS_PATH or AS4_PATH into the routing table, the possibility of loops caused by
these invalid paths is avoided.

The defined behaviour in RFC4893 and RFC4271 has significantly harmful effects
and it seems only by virtue of the fact that the implementations of many vendors
do not strictly comply with the RFCs that this problem has not had the same
impact for every vendor. At the current time, however, one cannot deploy a
4-byte capable Cisco IOS device, or an OpenBGP (current stable release) router
into the global table, without risking teardown of a every session via which a
global table is learnt.

Further discussion of this issue would be much appreciated, as a common and
consistent approach to rectifying the problem will benefit network operators far
more than individual vendor implementing their own solution. Should a consensus
be reached an update to the RFC is required in order to ensure that future
implementations do not exhibit this harmful behaviour.

Kind regards,
       Andy Davidson (NetSumo), andy.davidson@netsumo.com
       Jonathan Oddy (HostWay), jonathan.oddy@hostway.co.uk 
       Rob Shakir (GX Networks), rjs@eng.gxn.net

[0]: http://www.ietf.org/mail-archive/web/idr/current/msg03368.html
[1]: http://www.merit.edu/mail.archives/nanog/msg14345.html

Many thanks to David Freedman (Claranet) for assistance in developing the
recommendations in this document.


In addition to this - it looks like there's some fairly interesting coverage of another Juniper PSN at this blog.
So, if we take a moment to look at the following responses to questions that the leaders of the three parties involved in the "Digital Debate" on YouTube gave, concerning the Digital Economy Bill:
Sure, we get some tired old rhetoric, as expected. However, the key point here is that both Labour and the Conservatives appear to believe that they've done the right thing with this bill. Combine this with the fact that Gordon Brown mentioned a broadband rollout as part of the few ideas that he could during one of the debates, and I think that it starts to become apparently that politicians do not understand the UK "Internet" industry. Where there can be comments as to who is being "looked after" by each party (Cameron even mentions that the bill is most important for the media producers (or 'rights holders")) - I think the problem here is that politicians in the UK fundamentally do not understand how this industry operates. I think increasingly we are going to see the UK fall behind in terms of what we can roll-out due to impractical over-taxation, and ideas such as those put forward in the DEA.

Looking simply at two issues:
  1. Fibre Taxation - in the UK, if a business is to light up a fibre pair, as well as any standard taxes (e.g. VAT) that must be paid, then an additional VoA Business Rate is due on these fibres. This can be up to £500/pair/year outside of London, and £600/pair/year in the London metro region [source: Valuation Office Agency].

    Let's look at what this does for the telecommunications industry in the UK, especially for small players. Since such a company probably does not have a DWDM system, then the relatively cheap fibre runs are now taxed quite highly, should such a company then want to start increasing their capacity, then the additional costs are inflated due to taxation. Where larger players might be able to split these rates over a large number of DWDM channels (up to 32 or 64) a smaller provider might only have one channel - and hence the cost of infrastructure or customer links for smaller companies is inflated, due to the fact that they cannot justify the CapEx required for such multiplexing systems. Even for larger players, this isn't encouraging large scale fibre build out. If tax is paid per route-KM for every FTT{H,P,C} deployment, then this adds an additional overhead (in avoidable taxation!) to any such roll-out. Hardly an incentive for a commercial entity to begin such a deployment! Alongside the CapEx, OpEx, and business rates you are required to pay - the UK government will tax you just for lighting up the infrastructure they are encouraging you to build! This alone is not helping with any of the three party's plans for any kind of broadband roll-out, especially to rural areas where there is no profit for commercial entities to roll out such technologies.
  2. Digital Economy Act - Andrew Cormack of JANET (UK) gave an excellent presentation at UKNOF relating to the DEA. There are two key points here:
    • The government (and apparently the Tories) believe that this bill being pushed through in "wash-up" was the right thing to do. Contrast this with the fact that they also appear to be stating that the digital economy (and communications that such an economy provides) is key for Britain. I agree it's key, we're a services based economy, and if more services can be provided utilising the Internet, then one of two things will happen. Either the UK will not be equipped to deliver such services globally, and the "Digital Economy" will mean that these can then be out-sourced to other countries - or the UK will be in a position to grow the services that it can deliver, with the considerable skill of the UK workforce, into both global and European markets. Any bill therefore, that affects the manner in which this "Digital Economy" (by which I'm now referring to ISPs and telcos), should therefore, one would have thought, justify reasonable debate by the fully attended (?!) Commons!
    • Westminster appears to have no idea as to who they are legislating for. I am not against ensuring that the creative industries are able to protect their rights - however, this needs to be done in a manner that can be policed without damaging another industry. As Andrew said in his presentation the Government is unsure of how many ISPs are in scope - stating it could be 5, 10, 20 or 450. How can the impact of legislation be considered, if the Government cannot identify the scope? In addition, whilst many rights holders, I would imagine, will say "well, there is very little that is being requested of the ISPs here!" - the technical challenges of implementing mechanisms whereby specific IP addresses, and users can be located, within the timeframes that such complaints appear to take to be progressed, should be costed. I believe that most people within the xSP industry are not going to say "We don't care about your rights as a content producer", however, how can the Government expect our industry to pay directly to police this? We don't care that customer X is pulling data A, B and C - really, once it comes down to working in a larger ISP, we care about getting bit X to endpoint Z whilst ensuring any commercial guarantees that we have made for bit X.
    Another concern following these points is that it appears that very few of the UK ISP industry are being directly consulted here. Whilst there may be involvement - it's not something that I have seen mention of particularly amongst smaller ISPs in the community. Government should remember that legislation such as this affects all enterprises within this sector, and hence should consider them. The role of incumbents within this country already affects the delivery of many services, we don't need further legislation to push things further into their favour.
The reason I feel the need to mention this, is that it aggravates me whilst seeing responses such as the above. Politicians cannot absolve themselves of blame for such issues being pushed through in what I feel is quite an undemocratic manner. I'm still not sure who I am going to vote for - but as far as I see it, the huge lack of understanding of the industry within which I work will mean that whoever is in power during the next Parliament will likely not be in the right place to make legislation that actually takes into account how this industry works. Because of this, the UK's economy will suffer - which is a great shame.
Tagged in: Tech, ISP, UK
Leaving AS5413

For information, and because it means that I have revoked a bunch of UIDs from my GPG Key I am no longer working at AS5413 (Vialtus, Daisy, GX Networks etc.) as of June 18th, 2010. It's been a good two years, but the company direction no longer co-incides with the direction in which I would like to go. I've enjoyed the projects I've worked on, been in contact with a lot of great people, and learnt a lot!

As of Monday, I'll be taking on a IP and Data Network Design Role at Cable & Wireless.

Tagged in: Work, Me, ISP

Tom Bird of PortFast and Brandon Butterworth of Bogons do a great job of webcasting, and recording UKNOF video. Thanks to them, the video of the presentation I gave at UKNOF16 can be watched here. Or you can download it by clicking the image below!

As always, thoughts/comments/corrections most welcome!

This is also probably a good time to mention that my new work mail address is rob.shakir (at) cw.com


rjs@rob.sh sip:rjs@rob.sh
previous posts
contact details
gps logs
 
Fran Buckland [people]
hippy [people]
Andy Davidson [people]
Gem Atkinson [people]
rjs ssh key [tech]
rjs pgp key [tech]
londonfgss [cycling]
Rollapaluza [cycling]
CS Grupetto [cycling]
atom [rob.sh]
notebooks [rob.sh]
admin [rob.sh]
rss [rob.sh]
Stolen Bikes [london]
inhabitat [green]
core77 [design]