Modification required for Broadcom based FlexLOM cards (530/533FLR)

[halmartin]

The HPE 530FLR-SFP+ and 533FLR-T cards are based on the Broadcom BCM57810. These cards do not work with the adapter without modification.

The KCORES FlexibleLOM Adapter appears to have a more accurate pinout of the FlexLOM connector:

The following modifications to the adapter make it compatible with the Broadcom-based FlexibleLOM cards:

1. Cut the trace between Vdet (to make pin B1 N/C)
2. Cut the trace connecting PERST to pin A1 (between the center/right solder pads)
3. Bridge the PERST left/center solder pads so PERST is connected to A14
4. Bridge 3.3V pad to pin A1
5. Connect 3.3V pad to A10/A11
6. Bridge GND between pins A4/A5/A6/A7/A8

With the above changes, the 533FLR-T adapter is working for me: lspci output

---

Here is an annotated photo of the modifications required:

---

(The additional wire is to connect nWAKE)

[TobleMiner]

Cool! I'll prepare a RevB for testing. Many thanks for bringing this to my attention.

[halmartin]

For the pins requiring +3.3V, I bridged from +3.3V_AUX because it was easy and exposed on RevA, but I'm not sure if this is the "correct" way. KCORES are using +3.3V from pins B8/A9/A10 in their adapter.

I don't know enough KiCad to modify the adapter in this way, sorry.

[TobleMiner]

Hey, no worries. I'm not entirely sure about the 3.3V domains either. I wil derive PWROK from the main 3.3V rail since it is likely intended to be a power good signal for the 12V rail. To me another worry are the software deifned pins (SDP). I'm not entirely sure what they are doing. To me it seems a bit risky to pull them to 3.3V that hard.
Unfortunately I don't have access to any HP servers with a FlexibleLOM slot. Thus I can't verify whether the KCORES adapter matches what HP is doing.

[obedouet]

Hello,

I tested successfully the modification with a HP 530FLR-SFP+, without such hack, card was not visible through lspci. I did not add the additional wire. Thanks for the trick !

[MGX3D]

Thanks Tobias for the adapter and Hal for the mods! I wanted to confirm that revision A didn't work for me, but with the mods above I got 533FLR-T working. I haven't benchmarked it at 10GbE yet, but lspci/LnkSta says it's 5GT/s x8, ssh/scp works fine over it, etc.

However, the adapter still does not work for 561FLR-T (Intel X540), the only mod I didn't do is the nWAKE one, not sure if required?

One other suggestion I have for rev B is to reduce the copper plane that gets exposed when beveled: this might also be the reason JLCPCB doesn't bevel the PCBs. They didn't for me either, I did it manually and ended up exposing this plane and managed to kill a motherboard with it. I did notice the copper was exposed even though bevel was very light compared to other PCIe cards, but I figured I'd be safe once inserted because it should sit under the PCIe contacts in the connector, but I forgot to disconnect the PSU (rookie mistake) so the motherboard powered on while I was inserting the adapter and that was that! Luckily it was not a super expensive mb.

[TobleMiner]

Hi!

I'm super sorry to hear that. Retracting the GND plane a bit is something I should have done in the initial design. I've just redesigned the adapter, fixing both the pin assignments for the Flexible LOM slot (I've left out the SPD pins for now though) and the GND plane in the beveled area: https://github.com/TobleMiner/HPE-FlexibleLOM-adapter/tree/revb-untested

The design is not tested yet, but I've ordered it yesterday. So it should be here within the next two weeks or so.

[MGX3D]

Small update, I think I figured out why the rev A PCB (without any mods) worked with 561FLR-T for others but not for me initially: the reason seems to be as simple as using a PCIe powered video card vs one with separate power from PSU. I guess it has to do with total power draw of the two cards or possible mb quirks? Anyway, wanted to confirm that 561FLR-T without mods does work for me as well (when using a beefier video card with separate power connector).

[DrivebyCoding]

Unfortunately I don't have access to any HP servers with a FlexibleLOM slot. Thus I can't verify whether the KCORES adapter matches what HP is doing

I have a bunch of machines with FlexLOM and 544FLR. Anything in particular you want tested? Bricking a few MBs or HBAs is not a problem.

[halmartin]

@obedouet Did you add GND on pins B9 and B11?

I purchased a 530FLR-SFP+ that draws 2W and does not enumerate on the PCIe bus without grounding pins B9 and B11. Grounding pins B9 and B11 results in the card consuming 6W and successfully initializing.

[TobleMiner]

@halmartin Are you sure grounding B9 is required? According to the information I got that should only be TX enable for NC-SI.

[halmartin]

@TobleMiner I'm not certain, I have two versions of the modified adapter, one with the modifications I describe above, and the other with those modifications plus B9/B11 grounded.

The 530FLR-SFP+ I have does not work in the adapter that does not have B9/B11 grounded. Unfortunately my soldering iron has broken, so I cannot desolder connections to test if only B11 is required. I can test this when my new soldering iron arrives.

[obedouet]

@obedouet Did you add GND on pins B9 and B11?

I purchased a 530FLR-SFP+ that draws 2W and does not enumerate on the PCIe bus without grounding pins B9 and B11. Grounding pins B9 and B11 results in the card consuming 6W and successfully initializing.

Hello, I did not modify B9 and B11. I've only applied your steps 1-6. The card is working in a ESX since my comment (1/5) without any issue, really stable.

[dmg244]

/gerbers//revB_panel.zip have revA files inside.

[TobleMiner]

@dmg244 Oops, sorry for that. I've just updated the branch to now feature the RevB gerbers in that file.

[dmg244]

@TobleMiner Can you make revision C with the grounding of lines B9, B11, A4, A6, A7 and the connected WAKE line - A9 LOM to B11 PCI-E, A10 and A11 to +3.3V (all lines like KCORES design)? I want try to order this design for test. Why you use only +3.3V_AUX and don't use +3.3V?

[Nooby1]

I have ordered the RevB, I have some 530FLR-SFP+ (647581-B21) and QDR 544+FLR-QSFP (764286-B21)

I will update you in about two weeks

Does anyone know if these 544 cards are confirmed working in rev A?

[TobleMiner]

@dmg244

@TobleMiner Can you make revision C with the grounding of lines B9, B11, A4, A6, A7 and the connected WAKE line - A9 LOM to B11 PCI-E, A10 and A11 to +3.3V (all lines like KCORES design)? want try to order this design for test

Sure, will do. I don't feel that comfortable pulling the SDP pins that hard, but at least for a test it should be fine.

Why you use only +3.3V_AUX and don't use +3.3V?

By mistake. PWROK shoudl have been connected to the 3v3 power rail. However, I think B10 is supposed to be +3v3_aux to allow NC-SI to work even during system shutdown.

[TobleMiner]

@dmg244 Here you go: https://github.com/TobleMiner/HPE-FlexibleLOM-adapter/blob/revc-experimental/gerbers/revC_panel.zip

Think that should cover everything you might want to test. I've made sure to expose a good lenght of trace to any of the pins where we are not sure whether they are required for cards to enumerate or not. Feel free to experiment with cutting any of those traces and pulling them high/low or leaving them floating :)

[dmg244]

Thank you!
Already ordered from e-bay
2pcs HP 647579-001 649869-001 2-Port 530FLR SFP+ 10Gbps Ethernet Adapter
2pcs HPE 669281-001 Ethernet 560FLR-SFP+ 10GB 2-Ports PCI Express 2.0 x8 Adapter
HP 4-Port 684208-B21 629135-B21 634025-001 Ethernet 1Gb 331FLR Adapter

and connectors for testing.

JLCPCB will be produced PCB from 19 Feb

[0mnicy]

@dmg244 good luck with your testing. looking forward to seeing the results.

[Digifreakl]

I have ordered the RevB, I have some 530FLR-SFP+ (647581-B21) and QDR 544+FLR-QSFP (764286-B21)

I also bought some 530FLR-SFP+ (647581-B21), do you have any update?

[dmg244]

Hello! Thank you for you attention!

Latest news )))

Already received:
2pcs HPE 669281-001 Ethernet 560FLR-SFP+ 10GB 2-Ports PCI Express 2.0 x8 Adapter
2pcs HPE AJ838A LC/LC FC 30m Multi-mode OM3 Cable (491028-001)

still waiting PCB's and connectors.

Also still waiting:
2pcs HP 647579-001 649869-001 2-Port 530FLR SFP+ 10Gbps Ethernet Adapter
1pcs HP 4-Port 684208-B21 629135-B21 634025-001 Ethernet 1Gb 331FLR Adapter
2pcs Intel E10GSFPSR FTLX8571D3BCV-IT E65689-001 SFP+ Transceiver For X520-DA2/SR2
2pcs JD092B HPE X130 10G SFP+SR LC FTLX8571D3BCL-HZ 850nm for 5900AF

[Nooby1]

My PCBs arrived today. I got some soldered up and can confirm the revB worked with the 40gbe FLOM card QDR 544+FLR-QSFP (764286-B21)

I purchased 100 PCBs as the cost was not really much more than 20 from JLPCB due to the paneling of the design (required for edge chamfering with no extra cost) If anyone would like to buy some PCBS or completed cards off me do let me know. I am in the UK and will list on ebay also

[halmartin]

@Nooby1 Are you in the EU? I'd be interested in buying some PCBs from you.

[Nooby1]

@Nooby1 Are you in the EU? I'd be interested in buying some PCBs from you

Uk, and sure happy to sell up to 80 of them. I dont know the best way to share contact details on here however.

[halmartin]

Can you email me? username (at) gmail

[Rainfey]

@Nooby1 Is this offer still on the table?
Buying is preferable to waiting for shipping.

Regarding all the other adapters you ordered, have you managed to confirm if revB works with the 647579-001 model?

[Nooby1]

@Nooby1 Is this offer still on the table?
Buying is preferable to waiting for shipping.

Regarding all the other adapters you ordered, have you managed to confirm if revB works with the 647579-001 model?

Yes I still have pcb and completed models avalable, not tested with anything but connectx3 pro currently. Could test with 530flr in future

[Rainfey]

@Nooby1 I have added you to a project with my email address in the description - shame about github's lack of social features.
Drop me a line when you are able.

[dmg244]

Received all components.

I may confirm that adapter ver.C working with:

HPE 669281-001 Ethernet 560FLR-SFP+ 10GB 2-Ports PCI Express 2.0 x8 Adapter

#lspci
02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

HP 647579-001 649869-001 2-Port 530FLR SFP+ 10Gbps Ethernet Adapter

#lspci
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)

HP 4-Port 684208-B21 629135-B21 634025-001 Ethernet 1Gb 331FLR Adapter

#lspci
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

[kayjay010101]

@Nooby1 if you still have any complete boards I'd love to purchase one.

[NishaYume]

Hello @Nooby1, im interested in one if your version is compatible with HP 561FLR-T 10Gb.

Dont have the skills to solder this, hope someday somebody will sell the adapter ready to use!.

Anyway thanks!

[hps944]

Hello,

I found this project because I am searching a FlexLOM adapters for HPE 533FLR-T Cards, too.
Great work to found such a solution for this problem.
Is it still possible to buy such FlexLOM adapters with no soldering, just plug and play? I live Germany.
Maybe @Nooby1 would sell such Rev B type adapters, or does anyone know a source to buy it.

Thanks!

[halmartin]

Is it still possible to buy such FlexLOM adapters with no soldering, just plug and play? I live Germany.

Probably the reason no one is selling assembled adapters is because it is time consuming to assemble and test them, and people are generally purchasing FlexLOM cards because they're inexpensive compared to the PCIe equivalent.

Paying someone to assemble the adapter removes most/all of the value proposition, you would be better off just buying the PCIe version of the card.

You can buy unassembled Rev A adapters from sysmocom.

I have unassembled Rev B adapters that I'm willing to sell at cost, but you must solder them yourself.

[kayjay010101]

Hello,

I found this project because I am searching a FlexLOM adapters for HPE 533FLR-T Cards, too.
Great work to found such a solution for this problem.
Is it still possible to buy such FlexLOM adapters with no soldering, just plug and play? I live Germany.
Maybe @Nooby1 would sell such Rev B type adapters, or does anyone know a source to buy it.

Thanks!

I bought another adapter off eBay for quite cheap and to my surprise it does actually work. It does have a legit github page as well, albeit in Chinese. I have three of these adapters now and they all work on regular computers, making them + a fLOM card a much cheaper option than regular PCIe cards. Just search on eBay: "FlexibleLOM to PCIe" and it should be the only result.

[Nooby1]

I do have some rev B adapters left but @halmartin is correct, these take a long time to hand solder and QC check each one, checking it boots and enumerates. I cant send them out untested, and my time is expensive so I really dont make much money on each one for the time spent selling them at £20

[hps944]

I do have some rev B adapters left but @halmartin is correct, these take a long time to hand solder and QC check each one, checking it boots and enumerates. I cant send them out untested, and my time is expensive so I really dont make much money on each one for the time spent selling them at £20
Hello,
thanks for your offer. It is clear, that it is a lot of work and to spent a lot of time. Therefore the price is ok and I need the adapters. How can I get two adapters from you?

[hps944]

Is it still possible to buy such FlexLOM adapters with no soldering, just plug and play? I live Germany.

Probably the reason no one is selling assembled adapters is because it is time consuming to assemble and test them, and people are generally purchasing FlexLOM cards because they're inexpensive compared to the PCIe equivalent.

Paying someone to assemble the adapter removes most/all of the value proposition, you would be better off just buying the PCIe version of the card.

You can buy unassembled Rev A adapters from sysmocom.

I have unassembled Rev B adapters that I'm willing to sell at cost, but you must solder them yourself.

Hello,
thanks for your offer. I try to get 2 adapters from Nooby1, but I also want to try soldering them by myself. How can I buy 2 adapters?
Thank you.

[Nooby1]

@hps944 I have invited you to a FLOMSELL code repository where we can exchange details

[halmartin]

@hps944 My contact details can be found in my profile.

[halmartin]

For anyone looking to use a 530/533FLR with the Rev. B adapter, you still need to make a modification to connect A10 and A11 to 3.3V or the card will not work.

However, the modification is much easier than Rev. A:

[megabug360]

Just got my PCBs today and was testing rev A with mods and my 533FLR-T just blew up.

Thoughts?

Nevermind... had a short from PWROK to 12V+

EDIT2: Repaired my card, working fine now... thanks for this nice pcb.

[Blueblazer172]

Does REV.A work with a HP 647579-001 2-Port 530FLR SFP+ 10Gbps Ethernet Adapter? Do you know this?

[halmartin]

Does REV.A work with a HP 647579-001 2-Port 530FLR SFP+ 10Gbps Ethernet Adapter? Do you know this?

No, Rev. A adapters must be modified.

[ChristophDreier]

I just modified my Rev A adapter as shown in the original comment and the 560FLR-SFP+ is finally working. So for this card the mod is definitely needed too.

[SaturnusDJ]

Is rev B or C for sale anywhere or by anyone? I can try the soldering myself.

@Nooby1
What is this "FLOMSELL code repository"?

[halmartin]

Is rev B or C for sale anywhere or by anyone? I can try the soldering myself.

eBay is flooded with these adapters by now, and you can find an assembled adapter for ~$15, one such listing has the title "PCIE X8 to HP FlexibleLOM Quad GbE Ethernet 331FLR 366 530 544 561 FLR CX3 ALOM" and ships from China.

The predominant design available on eBay is the square KCORES one, which should work with Broadcom based cards without modification (see the first post, which is the schematic of the KCORES adapter and was the basis for rev B).

There are some people selling the TobleMiner FlexLOM adapter on eBay, however you run the risk of receiving a rev A PCB, which will require modification. I would stick to the KCORES design if you want something assembled and compatible with the Broadcom cards.

[SaturnusDJ]

@halmartin

Sorry, I completely forgot I did some research on that.

An acquaintance of me ordered from "hbr2015" (https://www.ebay.com/itm/144574303925) on eBay and found the adapter not aligning with the mounting of the computer case. I have not had my hands on, so I don't know exactly what's up.

So I messaged the seller "zeewill7" about https://www.ebay.com/itm/165425991830, which says to be a KCORES design on the PCB and he said they tested the 533FLR-T to be not working, otherwise they would have mentioned that model. I was very surprised, and that's why I am asking now.

[halmartin]

I don't have any KCORES adapters, so I cannot confirm or deny the issues you describe.

As I mentioned previously

For anyone looking to use a 530/533FLR with the Rev. B adapter, you still need to make a modification to connect A10 and A11 to 3.3V or the card will not work.

The schematic in the first post is from the KCORES adapter, and shows A10/11 connected to +3.3V.

This listing from zeewill7 claims support for 530FLR-SFP+. If the 530FLR-SFP+ works, the 533FLR-T should work as well.

[SaturnusDJ]

Thanks for that information. Might take the gamble then. Will see if any more information comes up next few days, then order.

[SaturnusDJ]

Just found out my x16 slot is only filled with copper for x4... Is that gonna be a problem to get the x8 card working at all or will it only be a bandwidth limitation?

(https://en.wikipedia.org/wiki/PCI_Express#History_and_revisions)

Edit: It should work. There are even x1 and x4 adapters available (which do limit bandwidth obviously). I will report back.

[halmartin]

Is that gonna be a problem to get the x8 card working at all or will it only be a bandwidth limitation?

It will only be a bandwidth limitation. Using kapton tape, I limited the adapter to PCIe x4:

01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 11)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 530FLR-SFP+ Adapter
        Physical Slot: 4
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        IOMMU group: 1
        Region 0: Memory at f5800000 (64-bit, prefetchable) [size=8M]
        Region 2: Memory at f5000000 (64-bit, prefetchable) [size=8M]
        Region 4: Memory at f6010000 (64-bit, prefetchable) [size=64K]
        Expansion ROM at f7a40000 [disabled] [size=256K]
        Capabilities: [48] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] Vital Product Data
                Product Name: HP Ethernet 10Gb 2-port 530FLR-SFP+ Adapter
                Read-only fields:
                        [PN] Part number: 647579-001
                        [EC] Engineering changes: A-5252
                        [MN] Manufacture ID: 103C
                        [V0] Vendor specific: 12W PCIeGen2
                        [V1] Vendor specific: 0.0.0.0
                        [V3] Vendor specific: 7.4.22
                        [V5] Vendor specific: 0A
                        [V6] Vendor specific: 7.4.15
                        [V7] Vendor specific: 530FLR-SFP+
                        [RV] Reserved: checksum good, 215 byte(s) reserved
                End
        Capabilities: [a0] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00001000
        Capabilities: [ac] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr+ NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x4 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [13c v1] Device Serial Number d8-9d-aa-bb-cc-dd-ee-ff
        Capabilities: [150 v1] Power Budgeting <?>
        Capabilities: [160 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
                        Status: NegoPending- InProgress-
        Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
                IOVSta: Migration-
                Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 8, stride: 1, Device ID: 16af
                Supported Page Size: 000005ff, System Page Size: 00000001
                Region 0: Memory at 00000000f6020000 (64-bit, prefetchable)
                Region 4: Memory at 00000000f60a0000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [220 v1] Physical Resizable BAR
                BAR 2: current size: 8MB, supported: 8MB
        Capabilities: [300 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Kernel driver in use: bnx2x
        Kernel modules: bnx2x

01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 11)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 530FLR-SFP+ Adapter
        Physical Slot: 4
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin B routed to IRQ 17
        IOMMU group: 1
        Region 0: Memory at f4800000 (64-bit, prefetchable) [size=8M]
        Region 2: Memory at f4000000 (64-bit, prefetchable) [size=8M]
        Region 4: Memory at f6000000 (64-bit, prefetchable) [size=64K]
        Expansion ROM at f7a00000 [disabled] [size=256K]
        Capabilities: [48] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] Vital Product Data
                Product Name: HP Ethernet 10Gb 2-port 530FLR-SFP+ Adapter
                Read-only fields:
                        [PN] Part number: 647579-001
                        [EC] Engineering changes: A-5252
                        [MN] Manufacture ID: 103C
                        [V0] Vendor specific: 12W PCIeGen2
                        [V1] Vendor specific: 0.0.0.0
                        [V3] Vendor specific: 7.4.22
                        [V5] Vendor specific: 0A
                        [V6] Vendor specific: 7.4.15
                        [V7] Vendor specific: 530FLR-SFP+
                        [RV] Reserved: checksum good, 215 byte(s) reserved
                End
        Capabilities: [a0] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00001000
        Capabilities: [ac] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr+ NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x4 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [13c v1] Device Serial Number d8-9d-aa-bb-cc-dd-ee-ff
        Capabilities: [150 v1] Power Budgeting <?>
        Capabilities: [160 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
                        Status: NegoPending- InProgress-
        Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
                IOVSta: Migration-
                Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 71, stride: 1, Device ID: 16af
                Supported Page Size: 000005ff, System Page Size: 00000001
                Region 0: Memory at 00000000f7a80000 (64-bit, prefetchable)
                Region 4: Memory at 00000000f60c0000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [220 v1] Physical Resizable BAR
                BAR 2: current size: 8MB, supported: 8MB
        Kernel driver in use: bnx2x
        Kernel modules: bnx2x

[SaturnusDJ]

Thanks! Is the negotiated link speed still 10Gbit/s when having capable devices on both ends? What actual speed do you reach? I think I calculated that when bonding two interfaces, they should reach 16Gbit/s on PCI-E x4 instead of 20Gbit/s on x8.

[halmartin]

Is the negotiated link speed still 10Gbit/s when having capable devices on both ends?

Yes, the PCIe lane count does not affect the negotiated Ethernet link speed.

What actual speed do you reach? I think I calculated that when bonding two interfaces, they should reach 16Gbit/s on PCI-E x4 instead of 20Gbit/s on x8.

I don't have the capability to run performance tests at the moment.

[SaturnusDJ]

Thanks for investigating! I will post here after building and testing. Unsure when. The adapters are still not shipped.

[SaturnusDJ]

zeewill7's adapter is working on two 533FLR-T cards for me. 10Gbit/s is being reached.

An issue though:
Having a normal ATX case, when having the perspective from the side panel, the PCIE slot of the adapter is placed a few mm too high, resulting in the card being slightly diagonally placed, in order to have the ports come out the back of the pc. This in turn blocks the screw hole of the next slot. Worse is that it is so diagonal that its own screw has difficulty securing it in place.

To solve this, the PCIE slot on the adapter should be slightly more biased towards the side containing the fan header and credits text, instead of being centred exactly above the FlexibleLOM slot.

Anyone else with this issue or could it be 5xxFLR-T only?

Now having the hardware, another issue is high pitch frequency coming from the card. Not maintainable to be near it. Another issue is that I cannot find any software to bond/team the interfaces on Windows 10. (Have not tried Linux yet.) Maybe my seller can get me 561FLR-T cards.

I have 3 extra adapters at this moment, after ordering 5.

[huseyinozsut]

Hi,

I need 10gbit network between my truenas server and my main pc. In my country the 10gbit used network cards (sfp+ and optic transmitter) are around 50 usd. This is extreme for my budget. But I found one 561FLR-T and one 533FLR-T 10gbit network cards for 20 usd each.

Since I am on a limited budget, I thought maybe I can cut some of the cables of cabled riser cards I already own and resolder them to use it as a FLR to PCI-E adapter.

The riser I am talking about is this:

As I understand the modification I need to do for 561FLR and 533FLR is different. Or can 561FLR work with the setup on top of this thread? If yes, then all I need to do is replicate the table at top with my riser cards. If not, is there a chart that I can read?